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Executive summary 


Executive summary 


The project 


The TextNow Transition Programme aimed to improve the reading comprehension skills of pupils at 
the transition from primary to secondary school by encouraging engagement in, and enjoyment of, 
reading. The programme was delivered by Unitas, a national charity that helps young people access, 
participate, and progress in mainstream education and training. 


Participating students received 20-minute one to one sessions with a volunteer coach each weekday 
for five weeks at the end of primary school and for a further 10 weeks at the start of secondary school. 
Children were expected to read independently for a further 20 minutes per day, and were rewarded for 
attendance with credits that could be used to buy books online. 


The trial examined the impact of the programme on 501 pupils in 96 schools across England who had 
been identified as unlikely to achieve Level 4a or above by the end of Key Stage 2. Pupils who were 
not likely to gain at least Level 2 were not included in the trial. 


The study was funded by the Education Endowment Foundation as one of 24 projects in a themed 
round on literacy catch-up at the primary-secondary transition. Projects funded within this round aimed 
to identify effective ways to support pupils not achieving Level 4 in English at the end of Key Stage 2. 
The project was one of four funded with a particular focus on reading for pleasure. 


1. The trial has not provided any evidence that the TextNow Transition Programme improved reading 
comprehension or attitudes towards reading for pleasure over the transition from primary to secondary 
school. 


2. On average, pupils who participated in the programme made slightly less progress than similar pupils who 
did not. However, this finding was not statistically significant, meaning that it could have occurred by 
chance. 


3. The programme was found to have a differential effect for pupils eligible for free school meals compared 
to their peers. A small positive (but not significant) effect was found for pupils eligible for free school 
meals, while a negative (and statistically significant) effect was detected for pupils not eligible for free 
school meals. It is unclear why this differential effect was found. 


4. Higher attendance at the 20-minute daily coaching sessions was found to have a positive impact on 
reading comprehension. However, this was only found to be statistically significant for attendance at 
sessions in secondary schools. Attendance of the coaching sessions was not found to have an impact on 
the secondary outcomes (liking reading and motivation to read). 


5. The programme appeared to be more effective when coaches were highly trained, enthusiastic and 
committed, and when secondary schools worked closely with feeder primaries to coordinate all elements 
of the programme. 
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What impact did it have? 


On average, pupils participating in the programme made less progress than those who did not. The 
size of the difference was small, and can be envisaged as saying that pupils who participated in the 
programme made approximately one months less progress than those who did not. However, the 
finding was not statistically significant, meaning that it is not possible to conclude with confidence that 
the observed effect was due to the programme rather than chance. 


The programme appeared to have a different impact on pupils eligible for and claiming free school 
meals compared to their peers. To look at this difference more closely, additional, separate analyses 
were conducted on pupils eligible for free school meals and their peers. This revealed a positive effect 
for pupils eligible for free school meals. However, this effect was not statistically significant, meaning 
that it could have occurred by chance. A negative effect equivalent to three months’ less progress was 
found for pupils not eligible for free school meals. The effect was found to be statistically significant, 
which means that is it unlikely to have occurred by chance. 


The programme did not appear to have a differential effect according to gender or prior attainment. 


: -0.06 
Intervention vs. Control 391 -1 month No £Ee 
(-0.22, +0.09) “=. 
Intervention vs. Control + 0.18 
11 2 month N 
(FSM only) 6 | (043, +0.48) | temants ° 
Intervention vs. Control -0.19 
275 -3 months Yes 
(Non-FSM only) (-0.01, -0.36) 
*For more information about evidence ratings, see Appendix VII in the main evaluation report. 
Evidence ratings are not provided for sub-group analyses which will always be less secure than overall findings 
*“For more information about cost ratings, see Appendix VIII in the main evaluation report. 


In addition to attainment, pupils’ enjoyment of reading and motivation to read was assessed. No 
statistically significant difference was detected between participating pupils and those in the 
comparison group on either measure. 


Overall attendance at sessions was good, but higher attendance rates at the 20-minute daily coaching 
sessions were found to have a positive impact on reading comprehension. However, this was only 
found to be statistically significant for attendance at sessions in secondary schools. The programme 
appeared to be more effective when secondary schools worked closely with feeder primaries to co- 
ordinate all elements of the programme. In addition, more effective coaches were highly trained, 
enthusiastic and committed. 


How secure is this finding? 


Overall, the evaluation findings are judged to be of moderate security. This assessment takes into 
account a number of factors including the study’s design, size and the level of drop-out. 
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This evaluation was set up as an effectiveness trial. Effectiveness trials aim to test the intervention 
under realistic conditions in a large number of schools. A randomised controlled trial design was 
employed to compare outcomes of pupils receiving the intervention to similar pupils who did not. 


This study had some limitations with regard to recruitment and attrition (e.g. where pupils dropped out 
of the intervention, or where test data for participating pupils was not available). 501 pupils were 
initially recruited compared to the desired sample size of 600 and of these, 391 provided full data for 
analysis of the primary outcome. However, despite this attrition, pre-test comparisons showed the 
comparison and intervention groups were well balanced. The study was large enough to detect an 
effect size of 0.2 (equivalent to approximately three months’ additional progress). This means that any 
impact at or above this level could have been detected with a high level of confidence. 


The sample of schools in the study was broadly comparable with other English schools with above 
average levels of disadvantage and thus the results are somewhat generalisable beyond the 
immediate context. 


Generally, the literature suggests there is potential in tutoring programmes for improving both 
academic and social and emotional outcomes. However, existing evidence also suggests that highly 
structured tutoring programmes provide the best effects on outcomes (i.e. where tutors are well 
trained, lessons focus on specific reading skills and sessions are timetabled systematically). This 
recommendation may be counter to the TextNow underpinning model of ‘choice — enjoyment — 
comprehension’. Furthermore, there is lack of systematic evidence in the wider academic literature for 
a pathway of causality between choice and comprehension. 


The study provides important insights into the feasibility of the programme in terms of: its potential 
effects on outcomes; an estimate of these potential effect sizes (providing valuable information on 
sample sizes for future study); the impacts on different groups (particularly with regard to free school 
meal eligibility); and key facilitators and barriers to implementation (from the process evaluation and 
the exploratory analysis of dosage). 


How much does it cost? 


Based on estimates from Unitas, the direct cost that schools would be required to pay for TextNow 
programme is £112 per pupil. This estimate includes resources (£50 per pupil), salary costs (£46), 
administration and other (£16). The estimate assumes a minimum cohort size of 10 and that sufficient 
schools are in the programme to make it viable. In addition, to run the TextNow programme each 
school must appoint a coordinator to recruit and manage volunteer coaches, manage programme 
delivery and undertake quality assurance and reporting. The programme also requires volunteer 
coach time of 20 minutes per pupil per day for 75 days, plus time for initial training, preparation for 
coaching sessions and keeping records. 


This estimate does not include estimates of the cost of co-ordinators, coaches and other resources in 
kind. 


Education Endowment Foundation 4 


Tahimeye (ULey (Cola) 


1. Introduction 


1.1 Intervention 


The TextNow Transition Programme, developed by national charity Unitas, is a reading for pleasure 
literacy programme with one to one sessions with a volunteer coach occurring 20 minutes a day each 
weekday. This occurs at two time points: in Year Six, the final year of primary school for five weeks in 
the second half of the summer term; and at the beginning of Year 7, the first year of secondary school, 
for ten weeks. The role of the coach is to support the three TextNow core principals of ‘choose, enjoy 
and understand’. Coaches are usually: teachers; teaching assistants; non-teaching staff such as 
technicians or administrative staff; parents; university students; community volunteers; or older year- 
group pupils (usually Years 10-13). A TextNow programme coordinator in each school recruits and 
supports the coaches and manages the administration of the programme. Coordinators are usually 
teachers, teaching assistants or school librarians. Schools are free to timetable the sessions whenever 
suits them best, for example through withdrawal from lessons, during registration or tutor periods, or in 
breaks, at lunchtime or after school. 


The programme also includes access to My Choice!, an online resource that has news, quizzes and 
competitions to engage young people in activities to do with reading, as well as giving them access to 
a book store. Participants are given credits to choose books from MyChoice! for summer holiday 
reading at the beginning of the project. They are able to earn further credits for their attendance and 
engagement with the coaching sessions during the five-week programme in Year 6 and the ten-week 
programme in Year 7: these can be used to ‘buy’ books for reading at home and to build their own 
personal library. 


This trial is the first time that the TextNow Transition Programme has been delivered. It was adapted 
from the Unitas TextNow programme that targets young people in their middle teenage years. 
TextNow was originally designed to meet the needs of young people in the youth justice system that 
were not in education, training or employment (NEET) and struggling with poor reading skills. 
Gradually TextNow was made available to the education sector, initially focusing on alternative 
education providers, but increasingly being used in mainstream secondary schools from 2010, when it 
was provided as a ten-week programme. At any one time there are normally between 25 and 35 
schools active in the secondary TextNow programme involving 400 to 600 pupils. During its 
development phase, the TextNow programme had working titles of ‘Intensive Reading Programme’ 
and ‘Reading Matters’, but these were insufficiently distinctive from other programmes, so the 
programme was named ‘TextNow in 2009. 


1.2 Background evidence 


There is a wealth of professional literature on literacy coaching (e.g. Blamey et al., 2008) and also 
quite widespread use of ‘literacy coaches’. The research on literacy coaches is almost entirely US- 
based, and in this context refers to specialised teachers — equivalent in status to literacy advisers, 
tutors or consultants in the UK context. In the US these coaches are often attached to a university 
reading laboratory or literacy research centre. The Unitas TextNow programme uses the word ‘coach’ 
in a more general sense as teachers, parents and peers who are involved in supporting pupils with 
their reading thus suggesting a more general ‘tutoring’ approach. 


As the Unitas TextNow programme involves a range of tutoring practices it is necessary to look at the 
evidence of effectiveness in this literature. According to a systematic review of RCT’s evaluating adult 
volunteer tutoring interventions (Ritter, Barnett, Denny, & Albin, 2009) it was found these programmes 


Education Endowment Foundation 5 


Tahineye (ULes (Cola) 


have a small but significant effect on pupils reading outcomes (hedges g = 0.23) including 
standandardised reading achievement scores e.g. SATs (g = 0.26), letters and words (g = 0.41), oral 
fluency (g = 0.30), and writing (g = 0.45). However, these programmes were not found to have 
significant effects on reading comprehension and mathematics outcomes. This systematic review also 
considered three implementation issues: (1) types of tutors; (2) age of tutees; and (3) highly structured 
versus unstructured programme delivery. Of these factors, only level of programme structure was 
found to have a significant influence on effectiveness. In this respect ‘highly structured’ refers to the 
following programme characteristics: tutors are well trained; lessons focus on specific reading skills; 
and sessions are timetabled systematically. In another review of effective programmes for struggling 
readers (Slavin et al., 2009), one to one volunteer tutoring was found to be an effective method for 
improving reading. The three main conclusions in this research were: (1) that tutoring focused on 
phonics was most effective; (2) teachers were more effective than teaching assistants; and (3) that 
one to one tutoring was more effective than group-based sessions. 


Randomised trials of peer tutoring (where pupils tutor other pupils) have also shown promising results. 
For example a large cluster randomised trial of 129 schools showed that peer tutoring could produce 
small effects (d = 0.20) on reading and mathematics outcomes (Tymms et al., 2011). However, these 
effects were only seen for cross-age tutoring (where older more experienced readers tutored younger 
readers). 


TextNow is an established programme from which the transition version in this study is developed. A 
previous evaluation of the established TextNow programme (Brooks, Tarling and Adams, 2011) 
showed moderate effect sizes on secondary school children’s reading outcomes (ES = 0.49) which are 
in fact relatively large compared to other tutoring programmes like those discussed above. However, 
this study was not a RCT and used standardised scores as a comparison group to the TextNow 
intervention group. As a result, there is likely to be bias in comparisons between the intervention group 
and the standardised sample. Furthermore, there may be within-group bias in that those who 
completed the full intervention as well both pre- and post-tests were initially significantly lower in 
reading age than those who did not complete all elements of the study. Regardless of these design 
limitations, which may have led to inflated or spurious effects, the study showed that the programme 
was more effective for older children (mean sample age 14 years 11 months) and for those children 
who received an increased input (more sessions) which is useful information in relation to programme 
implementation guidance. 


The TextNow programme utilises a ‘choose-enjoy-understand’ rubric. Like similar models in literacy 
education (e.g. ‘read-and-respond’ for comprehension, or ‘look-cover-write-check’ for spelling) it 
conveys the programme’s intent to foster independent reading, enjoyment and comprehension. This 
type of model is well supported in the general literature (e.g. Hall et al., 2013) and in policy, and has a 
strong ‘common sense’ appeal. With the exception of ‘understand’ (assuming that is taken as a cipher 
for comprehension) these attributes are hard to define and measure. What might ‘improved choosing’ 
look like? How might reading enjoyment be measured in a reliable way? Despite these measurement 
issues, there is a literature that suggests a link between choice, enjoyment and motivation to read 
(Flowerday & Schraw, 2000; Gambrell, 1996; Hunt, 1970; Sanacore, 1999). There is also literature 
that suggests a link between enjoyment and comprehension (Moss and Hendershot, 2002). However, 
this literature is often non-experimental and formed from the views of teachers and pupils. In addition, 
at least one experimental study shows no link between pupil choice and their engagement with 
reading (Flowerday, Schraw, & Stevens, 2004). So, given the inconclusive evidence around pupil 
choice and reading outcomes along with the systematic evidence in support of highly structured 
tutoring programmes (potentially diminishing choice), there remains a need to further investigate 
pathways of causality between ‘choice — enjoyment — comprehension’ particularly in the context of 
tutoring programmes. 


The young people in the study also have access to online activities on MyChoice! The evaluation team 
are not aware of any existing research on the benefits of supporting reading in this way, although it is 
increasingly common in online book circles and web-based resources such as those provided by 
Booktrust in the UK. Furthermore, the TextNow programme provides credits to buy books and this 
dimension of the work clearly relates to the notion that book ownership is beneficial (Clark & Poulton, 
2011). However, the direction of causality is ambiguous — reading enjoyment might well result in a 
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higher levels of book ownership, but book ownership may not be sufficient to promote reading 
enjoyment. 


Finally, the Unitas TextNow Transition Programme has been adapted to overlap with transfer from 
primary to secondary school. Considerable attention has been paid to the potentially negative effects 
of transition, particularly with respect to the apparent performance dip in transfer. The evaluation team 
is not aware of any quantitative research to support effects from an alleged ‘transition dip’, even 
though the notion has informed the discourse of policy-makers (e.g. DfE, 2013). However, there is 
some more general quantitative evidence of the impact of summer vacations. Cooper et al's (1996) 
meta-analyses of 13 American studies indicated that overall achievement test scores declined by 
about one month over the summer vacation. However, the decline in reading was less than for 
mathematics and while scores for reading comprehension decreased, scores for vocabulary and 
reading recognition increased. Importantly, this study indicates that social class is an important 
indicator of the extent to which a child experiences a reading dip over the summer. As Alexander and 
Entwhisle's (2007) analysis of Baltimore children's reading scores over the summer vacations in 
primary school also shows, children from low-income families experience a larger overall decline in 
reading scores than children from highe -income families. The difference is particularly striking for 
reading recogniition where children for higher-income families were found to make gains but children 
from low-income families experienced a dip (Cooper et al., 1996). The authors of both of these studies 
point to the greater availability of books in higher-income families as a potential explanation for their 
findings. There is also some qualitative work which suggests that regression or stagnation of learning 
may occur during transition and may be due to secondary teachers underestimating the capabilities of 
Year Seven students (Galton, Gray and Ruddock 1999 & 2000; Kirkpatrick, 1992). Further research 
suggests 16% of transition students feel unprepared for secondary school but only 3% of students 
remain worried about secondary school in the term after their transition (Evangalou et al., 2008). 


As indicated above, the TextNow Transition Programme is an adapted version of the Unitas TextNow 
programme. Therefore, the evaluation is set within three early phases of development as outlined by 
Campbell et al. (2007): 


e Phase 0: Theoretical development (why should this intervention work?). 

e Phase 1: Logic Modelling (how does it work?). 

e Phase 2: Exploratory or pilot trial (optimising trial measures and looking at potential effects 
overall and on sub-groups). 


This would map onto the Education Endowment Foundation system as an effectiveness trial.' 


The rationale for the present evaluation is to test rigorously whether the adapted TextNow Transition 
Programme has a measurable effect on students’ reading skills and comprehension, as well as on a 
small number of other secondary outcomes. 


The project was funded as part of a £10 million grant awarded by the Department for Education to the 
EEF for projects dedicated to literacy catch-up for pupils at the primary-secondary transition who do 
not achieve Level 4 in English by the end of Key Stage 2. 


1 Further information regarding EEF trial tyoes may be found here: 
http://educationendowmentfoundation.org.uk/uploads/pdf/EEF_approach_to_process_evaluation.pdf 
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1.3 Evaluation objectives 


The aim of this evaluation was to measure, through a randomised controlled trial, the impact of the 
Unitas TextNow Transition Programme on children who were unlikely to achieve Level 4a in reading 
by the end of Key Stage 2 (KS2). The process evaluation aimed to examine the implementation of the 
programme and capture the perceptions and experiences of key stakeholders engaged with the 
programme. 


As set out in the trial protocol and registration, the evaluation was designed to address the following 
questions: 


e What is the impact of the programme, at post-test (i.e. following the delivery of the 
programme), on a number of specific reading outcomes for participating children? 


e Is the programme having a differential impact on children depending on: 
o their gender? 
o their socio-economic status? 


e Does the impact of the programme vary significantly with any variations found in 
implementation? 


1.4 Project team 


Intervention team 


A dedicated Unitas programme manager led the implementation of the TextNow Transition 
Programme. This included recruiting schools, securing school and parental consent to the trial, 
supporting schools in implementing the programme, and accessing TextNow training and resources, 
monitoring implementation and agreeing test dates with the schools. A dedicated full-time 
administrator supported the separate management arrangements and there was a significant amount 
of line management support, for example from the Director of Operations. 


There were 53 coordinators recruited in primary schools and 31 recruited in secondary schools to take 
responsibility for implementing the TextNow Transition Programme in their own school or across two 
schools. The coordinators were generally school leaders, teachers, teaching assistants or support 
staff responsible for the school library/learning centre. There were 379 coaches recruited to support 
young people in primary school and 381 in secondary schools. The majority of coaches were teaching 
assistants, but they also included teachers, library/learning centre support staff, parents, student 
coaches from the secondary schools involved and higher education students. 


Evaluation team 


The evaluation was co-directed by Dr Bronwen Maxwell (SHU) (responsible for overall direction and 
process evaluation) and Professor Paul Connolly (QUB) (impact evaluation). The project directors 
were supported by Sean Demack (SHU), Dr Liam O’Hare (QUB) and Anna Stevens (SHU) for the 
impact evaluation, and by Lucy Clague (SHU) for the process evaluation and project management. 
Professor Guy Merchant (SHU) acted as expert literacy advisor to the project. 
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1.5 Ethical review 


The evaluation was approved by the ethics committees at Sheffield Hallam University and Queen’s 
University Belfast. Participating primary and secondary schools entered into a licence agreement with 
Unitas for use of the TextNow resources and signed a memorandum of understanding that set out 
their roles and responsibilities for the trial as well as the role and responsibilities of Unitas and the 
evaluators (Appendix VI). Opt-in consent was sought from parents/carers prior to trial. The 
parent/carer consent form (Appendix V), which covered both participation in the programme and the 
evaluation was distributed to parents through their child's primary school. Only children who had 
written parental/carer consent where included in the randomisation process. 
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2. Methodology 


2.1 Trial Design 


The intervention was delivered at the individual level. Randomisation was at the individual pupil level 
but blocked within schools (see Randomisation section for more detail). There were two trial groups 
only (i.e. control and intervention). Children in the control were given credits for MyChoice! after the 
trial. The design was chosen to provide a robust evaluation of the Unitas TextNow Transition 
Programme with regard to its effects on literacy outcomes. Analysis was conducted using multi-level 
models as the pupils were naturally clustered into primary and secondary schools. No post-hoc 
changes were made to the trial design due to recruitment issues. 


2.2 Eligibility 


Unitas recruited 34 secondary schools across England to take part in the trial, selecting schools where 
the proportion of pupils eligible for free school meals was above the national average and pupil 
performance below average. Full details of how schools were selected are given in Section 3.2. 
Across the secondary schools a total of 62 feeder primary schools took part in the trial. Eligible 
participants were drawn from pupils in Year 6 in primary school at the start of the intervention in June 
2013 who were making the transition to the 34 secondary schools involved in the study. Only pupils 
identified through teacher assessments as not likely to achieve Level 4 in reading, or who were likely 
to achieve a Level 4, 4b or 4c by the end of KS2 were eligible to participate. Pupils who were not likely 
to achieve a minimum of Level 2 were not included in the study due to their need for more extensive 
additional support. No minimum or maximum numbers of participants were placed on schools. 


Opt-in parental/carer consent was sought prior to randomisation. Primary schools organised the 
distribution and the collection of consent forms. 


2.3 Intervention 


The Unitas TextNow Transition Programme comprises one to one sessions with a volunteer coach 
and access to the MyChoice! online resource and book selection system. The programme is 
organised around secondary schools and their feeder primary schools. Implementation of the 
programme in each school (primary and secondary) is managed by a school coordinator. Most school 
coordinators were teachers, teaching assistants or school librarians/learning centre managers. The 
school coordinator liaises with, and is directly supported by, the Unitas programme manager. The 
school coordinator is responsible for: 


e identifying and preparing young people to participate; 
e recruiting and supporting coaches (for primary schools this may be done by the linked 
secondary school); 


e monitoring, reviewing and assessing young people's progress — this includes a short reading 
interview and an end of programme final review with the coordinator, coach and young 
person, as well as ongoing oversight of progress and addressing any issues such as 
attendance; 
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e maintaining quality assurance systems and reporting to Unitas; 


e setting up a starter library of books from MyChoice! 


School coordinators are expected to undertake four online training modules in preparation for their role 
comprising information, videos, supporting resources and training assessments. While there is no 
face-to-face training or monitoring of completion of training, school coordinators are also supported 
directly by Unitas by email and telephone throughout the set-up and delivery of the programme. 


Coaching is undertaken on a one to one basis, however one coach may support more than one young 
person. Coaches may be from a variety of backgrounds. Most frequently they are teachers; teaching 
assistants; non-teaching staff such as technicians or administrative staff; parents; university students; 
community volunteers; or older year-group pupils (usually Years 10-13). Coaches meet with their 
young person for 20 minutes each weekday for five weeks in the second half of the summer term 
when the pupils are in Year 6 in their primary schools; and 10 weeks at the beginning of Year 7 when 
pupils are in secondary school. For most young people a different coach works with them in their 
primary and secondary schools. 


Sessions are timetabled at times that suits the school: this may be in lesson time, before or after 
school or at lunchtime. The coaching role involves planning sessions, initially using the reading profile 
created by the school coordinator after the reading interview with the young person and as they 
establish a relationship with the young person, increasingly focusing in on the young person's interests 
and needs. Coaches also review and record the young person's progress. Coaching is focused around 
three strands — choosing texts, enjoying texts and understanding texts. ‘Choosing texts' focuses on 
improving young people's skills in choosing reading materials that they are interested in and which are 
suitable for their level of ability, giving them a sense of ownership about their choice. It is also 
designed to help young people to become familiar with environments where they can find the texts 
and to be relaxed in those environments. Related coaching activities include talking about book 
choice, developing strategies to assess a book choice, visiting a library and using MyChoice! 


To support young people in ‘enjoying texts' coaches are recommended to deploy a range of strategies 
and resources focused on shared reading: the use of ICT (such as e-readers and the internet), 
discussing related books and films, and using audio books. The 'Understanding texts' coaching strand 
aims to help young people understand what they read at a literal and inferential level and to talk, with 
understanding, about what they have read. Techniques included in the coaching framework to support 
understanding of texts include: skimming and scanning; kinaesthetic activities such as acting out a 
scene; KWL (what do | know, what do | want to find out, and what did | learn); visual representations 
such as a storyboard or book cover design; and reciprocal teaching involving summarising, asking 
questions, clarifying and predicting and embellishing. Coaches are expected to undertake four online 
training modules prior to working with their young person. 


A key feature of the TextNow Transition Programme is to provide rewards to young people for their 
participation. This is achieved through credits (worth 25p) being awarded for each day of attendance 
and participation with reading on TextNow. The credits are added to each young person’s account on 
MyChoice!, a web resource with an ‘online bookstore’ and book club which aims to support and 
encourage independent reading. MyChoice! includes questions to guide young people's choice of 
books, as well as news, quizzes and competitions to engage young people in fun activities to do with 
reading. Young people use the credits they 'earn' to 'buy' their own books and build their own personal 
library. In addition young people were given credits at the beginning of the trial to ‘buy’ books from 
MyChoice! over the summer holiday to sustain their engagement during transition. Coaches are 
expected to use MyChoice! at least once a week with their young person during the intervention at 
primary and secondary school. 


The control group was ‘business as usual’ during the trial and received no TextNow coaching. 
Following the post-test the control group was provided with credits to 'buy' books from the MyChoice! 
website. 
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The process evaluation indicates that there were significant fidelity issues in the implementation of the 
TextNow Transition Programme. Some pupils did not receive the full dosage of coaching and some of 
the young people who received peer coaching reported that that the coaching was poor quality. Some 
pupils had no or limited access to MyChoice! Fidelity was most problematic where schools lacked 
commitment to the programme. Due to the timing of the primary intervention (post SATs) some 
primary schools did not perceive any benefit to the school from their participation. Some secondary 
schools also lacked commitment to the programme, particularly when they had been recruited as part 
of a chain of schools. These issues are discussed in more detail in the process evaluation in Section 4 
below. 


2.4 Outcomes 


The primary outcome in the study was reading comprehension. The New Group Reading Test was 
used as a post-test reading ability measure and employs sentence completion and passage 
comprehension items®. Appendix | contains a statistical and graphical summary for this primary 
outcome. The New Group Reading Test was administered online through GL online testing software. 
Independent invigilators were sent to the schools to facilitate the testing process and a teacher in each 
school oversaw this process. The invigilators were blinded to the group status of the individual pupils. 
The KS2 reading score (standardised national school literacy assessment, taken from the spring 
census 2013 and accessed through the National Pupil Database) acted as a pre-test measure in the 
analysis models. 


The secondary outcome of interest was ‘reading for pleasure’ or children enjoying and fostering a love 
of books and reading. It was chosen because it was an intermediate attitudinal change that would lead 
to end-point outcomes in assessed literacy ability and a core aim of the TextNow programme. Two 
sub-scales from the Progress in International Reading Literacy Study (PIRLS) were used as a post- 
test reading attitudes measure related to reading for pleasure. The two scales were: 1. 'Students Like 
Reading’ Scale and 2. 'Students Motivated to Read' Scale. Appendix | provides a statistical summary 
of these two scales and the questionnaire items they were derived from. As with the primary outcome, 
standardised national school literacy assessments were used as a pre-test measure in the secondary 
outcome models. The resulting observed correlations between this pre-test and the two secondary 
outcome measures were fairly weak® and so the value of the pre-test measure was more limited for 
the secondary outcomes compared with the primary outcome. The secondary outcome tests were 
administered during the same session and under the same conditions as the primary outcome test. 


2.5 Sample size 


At the design stage, the sample size calculation for this individually randomised, multi-site block 
randomised, RCT used the following indicators:* 


e Probability: to detect effects with a probability <0.05. 


e Power: with a power of 0.8. 


? NGRT Information available at: http://www.gl-assessment.co.uk/products/new-group-reading-test. 
36=0.19 (KS2 v 'Students like Reading’); r = 0.08 (KS2 v 'Students Motivated to Read’). 
“ Calculation made using Optimal Design software: http://hlmsoft.net/od/ 
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e Effect size: with an effect size in the range of g = 0.2-0.3 (from previous literature). 
e Site size: with an average site size of 20 pupils. 


e Intra-Cluster Correlation (ICC) = 0.05 i.e. the proportion of variation that is ‘explained’ by 
clustering at the school level. At this design stage, clustering at the secondary school was 
used in the power calculations. 


e Covariate — with the proportion of explained variance by level 1 covariate being estimated to 
be 0.5 (Pearson's Correlation Coefficient). 


This calculation produced an estimate of number of sites (i.e. secondary schools) required to be 16-30 
depending on effect size in the range outlined above (i.e. 0.2-0.3). Therefore, the original design for 
this trial was based upon a large sample of pupils (N=600) from approximately 30 secondary schools. 
Although more than the target number of schools were recruited (34) there was a shortfall in the 
number of children entered into the randomisation process (N=501). This was due to there being fewer 
eligible pupils per secondary school than had been predicted and the non-return of consent forms by 
parents/carers. 


Following the collection and processing of data, the Minimum Detectable Effect Size (MDES) for the 
primary outcome (New Group Reading Test) was calculated more precisely. This was done by using 
the actual values of the ICC and the correlation between the pre- and post-test measures. Please note 
that two sets of calculations are provided. The first assumes clustering by secondary school (this was 
the approach used to estimate the MDES at the design stage — see above) whilst the second assumes 
clustering by primary school. The statistical analyses (multilevel models) presented in this report 
assume clustering at the primary school level because of the higher ICC. 


e 501 pupils at randomisation 

e 34 secondary schools, 62 primary schools 

e Average site size (pupils per school) at randomisation = 14.7 (secondary school); 8.1 (primary 
school) 

e Inter-class correlation = 0.048 (for secondary), 0.053(for primary) 

e Pre-post correlation = 0.54 


MDES estimates: 
Assuming clustering at secondary school level = 0.20 
Assuming clustering at primary school level = 0.19 


Despite the smaller than expected sample size, the randomisation worked well providing two balanced 
groups for analysis. 


2.6 Randomisation 


The randomisation was conducted at an individual level. The individuals were block randomised 
initially by school and then by reading level as indicated by teacher-predicted KS2 results to create 
pairs. 


A random number generator (provided by Stata v13) assigned a number between 0 and 1 to each 
member of a pair and the individual with the highest random number in each pair was assigned to the 
intervention group. In cases where uneven numbers of pupils were present in a school, the remaining 
individual would be assigned to the intervention group if they had a random number of above .5 and if 
they had a random number below 0.5 they were assigned to the control group. The randomisation was 
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conducted by a member of the independent evaluation team who had been blinded to any identifying 
school or pupil data. 


2.7 Analysis 


Multilevel analyses were conducted using the Stata v13 software that took account of how participants 
were clustered into primary schools at the beginning of the trial. Because this was a project around the 
transition of young people from Y6 to Y7, they were also clustered differently at the end of the trial in 
relation to the different secondary schools they entered. The main analysis was run twice to assess 
whether taking into account the clustering of young people by primary or secondary school made a 
difference. Both approaches resulted in similar findings. Moreover, the degree of clustering effects 
was more evident in relation to the young people’s attendance at primary school, as would be 
expected, and thus the main multilevel analysis was based on taking their clustering by primary school 
into account. The main analyses were conducted on an "intent to treat" basis following which an 
examination of 'dose' or fidelity was undertaken. This means that the assessment attempted to 
measure the impact of a school running the programme in real-world conditions, recognising that 
attendance or implementation may not be perfect. 


Binary dummy variables were used to identify the intervention (coded “1”) and control (coded “O”) 
group participants. Similarly, binary dummy variables were used for the FSM (“1”=FSM, “0”=non- 
FSM)° and gender (“1”=female, “O”=male) variables. A pre-test measure of Key Stage 2 reading score 
was also used within the analyses. 


The modelling was conducted in stages; first, a main effects stage followed by an interaction stage. 
The main effects models included the intervention / control dummy variable along with the KS2 pre- 
test measure, FSM and gender dummies. The main effects model assumes that any impact of the 
intervention is consistent across different participant subgroups (for example males and females; FSM 
and non-FSM participants): to explore whether this was a reasonable assumption, the three interaction 
terms were introduced one at a time. The interaction terms included were: FSM*intervention (isolating 
FSM participants who received the intervention); a gender*intervention (isolating females who 
received the intervention); and KS2 pre-test*intervention (isolating the pre-test scores of intervention 
group participants). If an interaction term was found to be statistically significant, a subgroup analysis 
was used to explore this in more depth. For example, if a gender*intervention interaction term was 
found to be statistically significant, separate models would be run for the male and female subsamples 
to explore this. 


Following the impact analyses, further analyses focused on intervention group participants to explore 
‘dosage’. In this context, 'dosage' is defined as the number of days a participant attended a TextNow 
session. Multilevel models explored whether attendance had an impact on the three outcome 
measures. For each outcome, two models were constructed: the first used an overall (primary and 
secondary school) attendance variable whilst the second used separate attendance details for the 
primary and secondary school sessions. 


° The FSM measure identifies young people who are eligible and claiming free school meals from 
young people who are not eligible or eligible and not claiming them. Although this measure is widely 
used as a proxy for the socio-economic status of young people, it should be noted that there is a 
known undercount of FSM claimants that is estimated at approximately 200,000 (or 2%) of all 4-15 
years olds in England (Iniesta-Martinez and Evans, 2012). Whilst FSM is a rather simple measure of 
socio-economic status and has this problem of inaccuracy, it is readily available and until better socio- 
economic detail is collected, remains likely to be the main tool for taking socio-economics into account 
in educational research in England. 
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2.8 Process evaluation methodology 


The process evaluation was designed to illuminate the implementation of the TextNow Transition 
Programme and assess fidelity. Key areas explored were pupil and school engagement, the quality of 
implementation of the intervention, perceived programme outcomes and adherence to the planned 
programme. Data was collected through a variety of activities at different time points as shown in 
Table 2.1. 


Table 2.1 Process evaluation data collection 


June to July 2013 5 weeks of TextNow intervention in primary schools 
September to November 2013 Surveys of all primary coordinators and coaches 
September to mid-December 2013 10 weeks of TextNow intervention in primary schools 
Mid-Dec 2013 to February 2014 Surveys of all secondary coordinators and coaches 
Jan 2014 Four pupil focus groups 


The survey of primary and secondary coaches covered: their background and training, the 
arrangements for coaching sessions, approaches to coaching, pupil engagement, behaviour and 
outcomes, as well as coaches' suggestions for improving the programme. The survey of primary and 
secondary coordinators covered: their background and training, the numbers of coaches they worked 
with, activities and tasks carried out in the coordination role and programme outcomes. Coaches and 
coordinators were also asked for suggestions to improve the programme. All surveys were conducted 
anonymously online and analysed by the evaluators using descriptive statistics for numerical data and 
thematic analysis of open questions. Links to the coordinator survey were provided directly to all 
coordinators, initially by the evaluators and then to boost the response by Unitas. Coordinators were 
asked to pass the links to the coach survey to coaches in their school. Response rates from both 
primary and secondary coordinators were high (primary: 64%; secondary: 68%). Although a lower 
response rate was achieved for coaches (primary: 11%; secondary 21%), a response from at least 
one coach was received from over half of the schools recruited to the trial (Table 2.2). 
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Table 2.2 Process evaluation survey response rate 


Primary coordinator 36° 64% 30 55.6 
Secondary coordinator 21° 68% 19 61.2 
Primary coach 42 11% 27 50.0 
Secondary coach 72 21% 18 58.1 


Four pupil focus groups were conducted by one researcher from the evaluation team after the end of 
the programme. A purposive sample of participating schools was selected for the focus groups to 
maximise variation in terms of geographical distribution, rural/urban school location, socio-economic 
profile, and type of governance. Of the six schools originally approached, two declined, one due to a 
forthcoming Ofsted inspection, and another did not have staff capacity to make the necessary 
arrangements for the visit. The four schools which agreed to participate in the focus groups varied in 
their degree of engagement with the TextNow programme. The TextNow coordinator in each school 
was asked to select 6 pupils from the intervention group that were representative of the range of pupils 
who had received the intervention in terms of gender, reading ability, attitudes towards reading, 
programme outcomes and English as a second language status (EAL). Due to pupil absence, there 
were fewer focus group pupils at one school. In total, 22 pupils (10 girls and 12 boys) from the 
intervention group participated in focus groups conducted in four secondary schools. The participant 
characteristics broadly matched those requested by the evaluators. Pupils were asked about their 
experiences of the programme at both their primary and secondary schools and how they thought 
participation in the programme had impacted their enjoyment of reading and reading skills. Each focus 
group was recorded and a thematic focus group report with illustrative quotes was prepared. 


® This equated to 34 individuals, with two working across two different primary schools. 
’ Two schools had responses from two coordinators. 
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3. Impact evaluation 


3.1 Timeline 


Recruitment of schools 


Pre-test data (Key Stage 2 finely graded reading point score) was 
collected from schools to identify eligible pupils (Note: KS2 data used for 
analysis made available to the research team from DfE in January 2014) 


Collection of parental consent 


Randomisation of pupils to intervention and control groups 


Commencement of five weeks of TextNow in primary schools 


Access to MyChoice! continued throughout the school holidays 


Commencement of ten weeks of TextNow in secondary schools 


Intervention finishes in secondary schools 


Post-test (standardised age score from the NGRTB test) collected 


Pupils (N=35) at two schools who was unable to arrange an earlier test 
were tested 


3.2 Participants 


Recruitment of schools was undertaken by Unitas who undertook a scoping exercise drawing on pupil 
performance and FSM data to identify secondary schools likely to have eligible pupils. Due to the short 
time period over which schools were to be recruited, the strategy was to approach secondary schools 
which were part of networks or chains which would provide routes into multiple schools or schools with 
which Unitas had a pre-existing relationship. Between October 2012 and December 2012 Unitas made 
telephone and/or email approaches to head teachers in selected schools (the number of schools 
contacted was not recorded) to gain senior leader support for participation. Where some interest was 
expressed, Unitas then explained the trial through meetings with head teachers/relevant staff and 
phone calls. This resulted in an ‘interested list' of 50 schools. Unitas communicated further information 
on the trial methodology in January 2013 via email, telephone and/or meetings. This was summarised 
in a Memoranda of Understanding for primary and secondary schools (Appendix V). At this point a 
number of schools withdrew. Reasons for withdrawal cited were that the evaluation was too onerous 
or that they were unwilling to take part in the RCT methodology of selection of young people. In 
addition, some schools revised downwards the number of pupils they felt they could work with. In 
some instances, although the secondary school had committed to the project, primary feeder schools 
could not be convinced to participate, so the secondary school was forced to withdraw. Unitas chose 
to recruit secondary schools rather than primary schools as they would undertake the bulk of delivery 
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of the programme, and would be in the best position to identify a cohort of eligible pupils from multiple 
feeder primary schools. Secondary schools, supported by the Unitas programme manager, brought 
their feeder primary schools into the trial. None of the secondary schools which elected to participate 
and were able to bring on board their feeder primary schools had previously worked with Unitas or had 
run the secondary TextNow programme. At the point of randomisation 34 secondary schools (four 
above the original target number) across England and 62 primary schools were participating in the 
trial. 


All pupils in Year 6 in the feeder primary schools of the 34 participating secondary schools were 
assessed for eligibility by the secondary schools. Potential eligible pupils were identified by the 
secondary schools based on primary teachers' assessments of predicted reading point score at KS2. 
Parental/carer consent was requested for all pupils who were not predicted to achieve Level 4, or who 
were likely to achieve a fragile Level 4 (4b or 4c) and were likely to gain at least a Level 2. 


Only data on the 501 pupils that schools identified as eligible, and for whom schools had collected 
parental/carer consent, were passed to the evaluators. It is therefore not possible to compare those 
pupils whose parents did not give consent with the pupils included in the trial in terms of key 
characteristics. The number of pupils who met the pilot selection criteria and accepted a place at the 
participating secondary school was far below the initial prediction given by the secondary schools, 
resulting in an under-recruitment of pupils for the pilot (target number 600). The under-recruitment of 
pupils was further exacerbated by the process of gaining parental/carer consent. Issues included: the 
timing of the consent request clashing with the Easter holiday; some schools not adhering to the 
processes for distribution of consent forms requested by Unitas and/or not chasing missing forms; and 
the complexity of the form (Appendix V). 


As the flowchart (Figure 3.1) illustrates, 501 eligible pupils were entered into the trial. Of the 501 pupils 
who were randomised, 252 were allocated to the intervention group and 249 to the control group. 
Three of the control group pupils were given the intervention in primary school but were not given the 
intervention in their secondary school. Nineteen pupils in four primary schools in the intervention group 
did not receive the five-week intervention in their primary school and did not have access to 
MyChoice! over the summer, but did receive the intervention in the secondary school.® 


Of the 252 pupils allocated to the intervention group, 53 were lost to follow up: 32 of these 53 were in 
the five secondary schools that discontinued the programme and dropped out of the trial. Four of the 
five secondary schools did not start the 10-week secondary programme. Reasons given included: that: 
their feeder primaries had dropped out and not run the 5-week programme; logistical reasons making 
it impossible to implement the programme in the timescale required; and in one school, an impending 
Ofsted inspection. These four schools were unwilling, or did not have the capacity, to arrange the end- 
tests. The drop-out of these schools does not compromise the internal validity of the study but does 
have minor implications for the external validity or representativeness of the results for schools in 
general. Schools which were less engaged with the programme also tended to drop-out of the testing, 
meaning that the treatment effect is estimated on more engaged schools. The fifth secondary school 
dropped out in week eight of the ten-week secondary programme citing a lack of capacity to either 
complete the programme or arrange the end-test. Twenty-one of the 53 pupils lost to follow-up did not 
attend the end-test in their school. In total 199 intervention pupils were entered into the analysis of the 
primary outcome. Of the 249 pupils allocated to the control group, 56 were lost to follow-up, 28 in the 
secondary schools who withdrew from the trial and 28 who did not attend the end-test in their school. 
We were unable to match one of the control pupils to the NPD, so a total of 192 pupils from the control 
group were entered into the analysis of the primary outcome. In total pupils from 53 primary schools 
and 29 secondary schools were entered into the analysis. 


8 Of these 19 pupils, seven were lost to follow- up due to their secondary school withdrawing from the 
trial near the end of the ten- week programme in the secondary school. 
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The number of pupils entered into the analysis for the secondary outcomes was lower than for the 
primary outcome of reading comprehension (primary outcomes n=391; secondary outcome ‘Students 
Like Reading n=331; secondary outcome ‘Students Motivated to Read' n=352). This was due to 
missing responses relating to the questionnaire components used to construct the scales for the 
secondary outcomes. 


Figure 3.1: Trial participants 


Randomised (n=501) 
Secondary Schools = 34 
Primary Schools = 62 


Allocated to control (n=249) 
Allocated to intervention (n=252) 


Received allocated intervention (n=249) 
Received allocated intervention (n=252) 


Did not received allocated intervention (n=0) Did not received allocated intervention (n=0) 


Lost to follow-up (n=53) Lost to follow-up (n=56) 


Discontinued intervention (n=0) Foll Discontinued intervention (n=0) 


Unable to match to NPD (n=1) 


Schools lost to 
follow up 


Secondary = 5 
Primary = 9 


Analysed (n=199) Analysed (n=192) 


Excluded from analysis (n=0) iis Excluded from analysis (n=0) 


Schools included: 


Secondary = 29 
Primary = 53 
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3.3 School characteristics 


In terms of percentage of pupils with SEN, the average percentage for the sample was slightly higher 
than the national average, this was also the case for the percentage of pupils with English not as a 
first language. Compared to the national average, the schools sampled for the study had a higher 
average percentage of pupils eligible for free school meals and a lower average percentage of pupils 
achieving 5+ A*-C or equivalent GCSEs including A*-C in both English and Mathematics. Comparing 
the original sample to the achieved sample used in analysis reveals little difference between these 
sample characteristics (Table 3.1). 


Table 3.1 Secondary school characteristics of sample compared with national data 


Percentage of pupils with SEN 1.9% 3.1% 3.2% 
Percentage of pupils with English not as a first 13.6% 16.8% 16.1% 
language 

Percentage of pupils eligible for free school 16.3% 25.4% 25.4% 
meals 

Percentage of pupils achieving 5+ A*-C or 60.6% 51.4% 51.7% 
equivalents including A*-C in both English and 

Mathematics GCSEs 


Around half the schools in the original sample had an Ofsted rating of ‘Good’ whilst 9 had a rating of 
‘Requires improvement’. Three schools had an ‘Outstanding’ rating whilst 4 were deemed 
‘Inadequate’. The distribution of Ofsted ratings was similar across the sample used for analysis (Table 
3.2). 


Table 3.2 Secondary schools Ofsted ratings 


Outstanding 3 3 
Good 15 13 
Requires 9 7 
improvement 

Inadequate 4 3 


*Ofsted reports unavailable for 3 schools 
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The majority of schools in the original sample were Academy sponsor-led schools (n=18). In total 9 
schools were community schools, 4 were foundation schools, 2 were academy convertors and 1 was a 
Free School. Four of the 5 schools who dropped out of the trial were Academy sponsor-led schools 
(Table 3.3). 


Table 3.3 Secondary school type 


Original Sample for 
sample (n=34) analysis 
(n=29) 
Foundation School 4 3 
Community school 9 9 
Academy Converter Mainstream 2 2 
Academy Sponsor-Led 18 14 
Free School Mainstream 1 1 


As noted above, the schools in the sample were mostly in the upper range in terms of percentage of 
pupils eligible for free school meals; 21 schools were in the upper quintile (Table 3.4). 


Table 3.4 Pupils eligible for FSM 


FSM quintiles Original sample (n=34)* | Sample for analysis (n=29)* 
Lowest quintile 0 0 

Lower quintile 3 2 

Middle quintile 7 6 

Upper quintile 21 19 

Highest quintile 2 1 


* One school is not included in the 2013 KS4 Performance Tables 


The majority of schools (19) were in the lower quintile for attainment (Table 3.5). Although all drop-out 
schools were in this category, all of these were at the top end of the lower quintile. Hence with regards 
to the average percentage of pupils achieving 5+ A*-C or equivalents including A*-C in both English 
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and mathematics GCSEs little difference can be seen between the original sample and the achieved 
sample (Table 3.1). 


Table 3.5 Secondary school attainment 


Percentage of pupils Original sample Sample for analysis 
achieving 5+ A*—C or (n=34)* (n=29)* 
equivalents including A*—C 

in both English and 

Mathematics GCSEs 

quintiles 

Lowest quintile 2 2 
Lower quintile 19** 14 
Middle quintile 7 7 
Upper quintile 3 3 
Highest quintile 2 2 


* One school is not included in the 2013 KS4 Performance Tables 


= Although all drop-out schools appear in the lower quintile, these schools were close to the top end of this quintile, and the 
mean % attainment for the original sample is close to that of the analysis sample. 


3.4 Pupil characteristics 


Table 3.6 Number of participants involved in the TextNow Transition Programme RCT 
evaluation 


Intervention Control Intervention Control 
Participants 252 249° 199 192 
Primary Schools 62 62 53 53 


° It was not possible to match two of the control group pupils to the NPD; therefore the analysis of pre- 
test pupil characteristics is based on 247 control group pupils. 
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Secondary Schools 


34 


34 


29 


29 


In terms of FSM, gender and the pre-test measure, randomisation resulted in intervention and control 
group samples that were comparable at baseline. The samples remained statistically comparable at 
the post-test stage suggesting that the small attrition did not introduce bias in terms of FSM, gender 
and the KS2 pre-test measure (Tables 3.6-3.9). 


Table 3.7 FSM statistical snapshots at baseline and outcome 


Intervention 


Control 


Intervention 


Control 


Eligible and registered / 
claiming 


79 (31%) 


76 (31%) 


57 (29%) 


59 (31%) 


Not eligible or eligible 
and not registered / 
claiming 


173 (69%) 


171 (69%) 


142 (71%) 


133 (69%) 


Total 


252 (100%) 


247 (100%) 


199 (100%) 


192 (100%) 


Value (d.f.) 


p= 


Value (d.f.) 


Pearson chi-square test 


0.02 (1) 


0.889 


0.20 (1) 


0.65 


Table 3.8 Gender statistical snapshots at baseline and outcome 


Intervention Control Intervention Control 
Female 116 (46%) 114 (46%) 96 (48%) 86 (45%) 
Male 136 (54%) 133 (54%) 103 (52%) 106 (55%) 
Total 252 (100%) 247 (100%) 199 (100%) 192 (100%) 
Value (d.f.) p= Value (d.f.) p= 
Pearson chi-square | 0.001 (1) 0.978 0.47 (1) 0.49 
test 
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Intervention Control Intervention Control 
Mean (sd) 24.7 (4.16) 24.4 (4.16) 24.7 (4.29) 24.3 (4.27) 
Median (IQR) 25.4 (5.04) 25.1 (6.18) 25.4 (5.04) 25.1 (6.18) 
N 252 247 199 192 

mean difference p= mean difference |p 

(d.f.) (d.f.) 
t-test (df) +0.28 (497) 0.46 +0.34 (389) 0.43 
Effect Size (Hedges | +0.07 +0.08 
9) 
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Table 3.10 Primary outcome—reading comprehension" 
(n=391 participants in 53 primary schools) 


Impact evaluation 


Pre-test Gender FSM 
interaction Interaction Interaction 
coef. s.e. coef. s.e. coef. s.e. coef. s.e. 
Group (Intervention) -0.72 0.874 -0.72 0.868 -1.58 1.221 -1.96 1.043 
Pre-test (KS2 Reading) 1.54 0.111 1.33 0.152 1.54 0.111 1.54 0.110 
Interaction (Pre- | - - 0.40 0.206 - - - - 
test*Intervention) 
Gender (female) 1.58 0.926 1.59 0.921 0.65 1.309 1.75 0.924 
Interaction (gender*Intervention) | - - - - 1.84 1.834 - - 
FSM (eligible & claiming) -1.35 1.018 -1.31 1.013 -1.23 1.023 -3.42 1.408 
Interaction (FSM*Intervention) - - - - - - 4.41 1.948 
Constant 88.81 1.000 88.76 1.000 89.20 1.073 89.3 1.033 
7 

-2 Log Likelihood -1497.891 -1418.657 -1420.047 -1418.310 
School level variance (s.e.) 17.68 (6.239) 18.07 (6.309) 17.84 (6.259) 17.92 (6.233) 


Pupil level variance (s.e.) 


73.39 (5.721) 


72.48 (5.652) 


73.13 (5.700) 


72.39 (5.640) 


Intra-Class Correlation (s.e.)* 


0.19 (0.059) 


0.20 (0.060) 


0.20 (0.059) 


0.20 (0.059) 


'° See Appendix II for the approach to modelling and the key terms used in the tables in this section. 
= Reading comprehension was assessed using the GL New Group Reading Test. 
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Hedges g effect size’* for main effects model (95% Confidence Intervals): = -0.06 (-0.22, +0.09) 


A small negative effect size was found but this was not statistically significant and so we must 
conclude that there is no evidence of impact on the primary outcome. 


The FSM*intervention interaction term, however, was found to be positive and statistically significant, 
and when this is included into the model the (negative) FSM main effects term becomes statistically 
significant. This suggests that the TextNow Transition Programme had a different impact for FSM and 
non-FSM participants and to look more closely at this, models were re-run on the FSM and non-FSM 
subsamples. 


The pre-test KS2 measure was statistically significant in all models, reflecting the correlation between 
this test score and the primary outcome (Pearson's r=0.5). 


The following tables (3.11 and 3.12) summarise the primary outcome models for the FSM and non- 
FSM participant subsamples. 


Table 3.11 Primary outcome — reading comprehension 
(FSM Participant only, n=116 participants in 44 primary schools) 


Pre-test Gender 
interaction Interaction 
coef. s.e. coef. s.e. coef. s.e. 

Group (Intervention) 2.01 1.767 2.17 1.780 3.03 2.601 
Pre-test (KS2 Reading) 1.33 0.238 1.15 0.334 1.35 0.240 
Interaction (Pre-test*Intervention) — - ; 0.35 0.451 ; - 
Gender (female) 1.08 1.866 0.90 1.873 2.03 2.586 
Interaction (gender*Intervention) - - - - -1.99 3.718 
Constant 86.11 1.775 86.12 1.766 85.55 2.055 
-2 Log Likelihood -430.804 -430.509 -430.663 


"? For detail on how the Hedges g effect size was calculated see Appendix III. 
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School level variance (s.é.) 


19.38 (12.344) 


17.99 (12.043) 


Impact evaluation 


18.67 (12.276) 


Pupil level variance (s.e.) 


82.82 (13.477) 


83.26 (13.530) 


83.06 (13.540) 


Intra-Class Correlation (s.e.) 


0.19 (0.111) 


0.18 (0.110) 


0.18 (0.111) 


Hedges g effect size for main effects model (95% Confidence Intervals): +0.18 (-0.13, +0.48) 


Amongst the FSM subsample, a positive effect size was found. However, this does not reach 
statistical significance and so we conclude no impact. 


Table 3.12 Primary outcome — reading comprehension 
(Non-FSM participants only, n=275 participants in 52 primary schools) 


Pre-test Gender 
interaction Interaction 
coef. s.e. coef. s.e. coef. s.e. 

Group (Intervention) -2.08 0.999 = -2.16 0.990 = -3.35 1.350 
Pre-test (KS2 Reading) 1.56 0.121 1.34 0.166 1.57 0.120 
Interaction (Pre-test* Intervention) - - 0.45 0.231 - - 
Gender (female) 2.09 1.077 2.23 1.071 0.59 1.519 
Interaction (gender*Intervention) - - - - 2.91 2.085 
Constant 89.27 1.030 89.19 1.032 89.85 1.107 
-2 Log Likelihood -986.895 -984.983 -985.927 


School level variance (s.é.) 


17.79 (6.652) 


18.61 (6.816) 


17.78 (6.637) 


Pupil level variance (s.e.) 


65.42 (6.218) 


64.07 (6.100) 


64.91 (6.171) 
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Intra-Class Correlation (s.e.) 0.21 (0.068) 0.23 (0.070) 0.22 (0.068) 


Hedges g effect size for main effects model (95% Confidence Intervals): - 0.19 (-0.36, -0.01) 
Amongst the non-FSM subsample, a statistically significant negative effect size was found. 


Overall, the findings for FSM participants are encouraging but the evidence is insufficient to claim a 
positive impact whilst for non-FSM participants, it seems that the TextNow programme had a negative 
impact that was small but statistically significant. 


The analyses then focused on just the intervention group participants to look at whether attendance at 
the TextNow Transition Programme coaching sessions had an impact on the primary outcome 
measure. The attendance 'dosage' was approximated using attendance details provided by Unitas for 
all participating pupils. The time in days spent involved in the (20 minute) coaching one to one 
sessions was selected as the measure. The time spent in primary school and secondary school was 
kept separate in the analyses. Table 3.13 summarises these dosage measures for the 199 
intervention group participants. 


Table 3.13 Attendance dosage (weeks and days) — intervention group only (n=199) 


Weeks Days Weeks Days 
Expected dose 5 5x5=25 10 10x5=50 
Mean (sd) 4.4 (1.61) 17.8 (7.11) 9.9 (1.24) 42. (6.901) 
Median (IQR) 5.0 (0.00) 20.0 (6.00) 10.0 (0.00) 44.0 (10.00) 
Min 0 0 0 0 
Max 6 25 13 52 


Table 3.14 Primary outcome — reading comprehension — dosage analysis 
(Intervention sample only, n=199, across 53 primary schools) 


coef. s.e. coef. s.e. 


Pre-test (KS2 Reading) 1.69 0.152 1.67 0.150 
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Dosage Variables 

Days attended (overall) 0.09 0.061 - - 
Days attended (primary school) = - -0.05 0.106 
Days attended (secondary school) - - 0.15 0.072 
Constant 88.42 0.815 88.42 0.803 
-2 Log Likelihood -722.59 -721.26 

School level variance (s.e.) 14.07 (8.380) 13.29 (7.931) 


Pupil level variance (s.e.) 


72.49 (8.867) 


71.89 (8.711) 


Intra-Class Correlation (s.e.) 


0.16 (0.089) 


0.16 (0.086) 


Impact evaluation 


A statistically significant coefficient does emerge around attendance at the secondary school stage of 
the intervention. Increased time (in days) spent engaged with the TextNow intervention during Y7 


(secondary school) is associated with an increased score on the primary outcome. Time spent 


engaged with TextNow at the end of Y6 does seem to have an impact on the primary outcome. 


Table 3.15 Secondary outcome - ‘Students Like Reading'’® 
(n=331 participants in 53 primary schools) 


Pre-test Gender FSM 
interaction Interaction Interaction 
coef. s.e. coef. s.e. coef. s.e. coef. s.e. 
Group (Intervention) -0.10 0.584 -0.10 0.584 -0.51 0.804 -0.16 0.700 


'S This was measured using the Progress in International Reading Study (PIRLS) 2011 'Students Like 


Reading’ Scale. 
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Pre-test (KS2 Reading) 0.26 0.070 0.23 0.100 0.26 0.071 0.26 0.070 
Interaction (Pre-test*Intervention) - - 0.06 0.138 - - - - 
Gender (female) 3.17 0.596 3.19 0.597 2.70 0.866 3.18 0.599 
Interaction (gender*Intervention) - - - - 0.90 1.199 | - - 
FSM (eligible & claiming) 0.74 0.654 = 0.75 0.654 0.80 0.657 390.64 0.926 
Interaction (FSM*Intervention) - - - - - - 0.21 1.288 
Constant 12.93 0.561 12.92 0.562 13.13 0.617 12.96 0.590 
-2 Log Likelihood -1026.459 -1026.369 -1026.175 -1026.445 
School level variance (s.é.) 1.50 (1.016) 1.51 (1.017) 1.53 (1.024) 1.50 (1.016) 


Pupil level variance (s.e.) 


27.63 (2.274) 


27.61 (2.272) 


27.56 (2.269) 


27.63 (2.274) 


Intra-Class Correlation (s.e.)* 


0.05 (0.034) 


0.05 (0.034) 


0.05 (0.035) 


0.05 (0.034) 


Hedges g effect size for main effects model (95% Confidence Intervals): = -0.02 (-0.22, +0.18) 


A very small negative effect size was found but this is not statistically significant and so we must 
conclude that there is no evidence of impact on the secondary outcome ‘Students Like Reading’. No 
evidence was found for differential effects across participant subsamples (indicated by a lack of 
statistically significant interaction terms). The only statistically significant terms were (a) the KS2 pre- 
test measure, reflecting its correlation with the PIRLS 2011 ‘Students Like Reading’ outcome 
(Pearson's r=0.19) and (b) participants' gender (females had a significantly higher outcome score, 
although this was found to be independent of participation in the TextNow programme). 
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Table 3.16 Secondary outcome — ‘Students Like Reading’ scale — dosage analysis 
(Intervention sample only, n=173, across 53 primary schools) 


coef. s.e. coef. s.e. 
Pre-test (KS2 Reading) 0.26 0.105 0.26 0.105 
Dosage Variables 
Days attended (overall) 0.00 0.037 - - 
Days attended (primary school) = - 0.03 0.059 
Days attended (secondary school) - - -0.03 0.045 
Constant 14.53 0.445 14.53 0.444 
-2 Log Likelihood -550.47 -550.14 
School level variance (s.é.) 0.00 (0.000) 0.00 (0.000) 
Pupil level variance (s.e.) 33.99 (3.654) 33.86 (3.640) 
Intra-Class Correlation (s.e.) 0.00 (0.000) 0.00 (0.000) 


There is no evidence of attendance having an impact on the ‘Students Like Reading’ secondary 
outcome measure. 
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Table 3.17 Secondary outcome—'Students Motivated to Read’ scale” 
(n=352 participants in 53 primary schools) 


Pre-test Gender FSM 
interaction Interaction Interaction 
coef. s.e. coef. s.e. coef. s.e. coef. s.e. 

Group (Intervention) 0.24 0.388 0.23 0.388 0.39 0.540 0.14 0.461 
Pre-test (KS2 Reading) 0.01 0.047 0.04 0.067 0.06 0.047 0.06 0.047 
Interaction (Pre-test*Intervention) 7 ; 0.04 0.092 - 7 . - 
Gender (female) 0.69 0.398 0.69 0.398 0.85 0.575 0.70 0.399 
Interaction (gender*Intervention) - 3 - 5 -0.31 0.797 | - 3 
FSM (eligible & claiming) -0.33 0.439 -0.32 0.439 -0.34 0.441 -0.50 0.628 
Interaction (FSM*Intervention) - - - - - - 0.34 0.869 
Constant 13.05 1.061 13.04 1.060 13.06 1.063 13.05 1.061 
-2 Log Likelihood -959.635 -959.522 -959.560 -959.558 
School level variance (s.e.) 0.72 (0.550) 0.72 (0.550) 0.70 (0.548) 0.71 (0.548) 
Pupil level variance (s.e.) 13.05 (1.061) 13.04 (1.060) 13.06 (1.063) 13.05 (1.061) 
Intra-Class Correlation (s.e.)* 0.05 (0.039) 0.05 (0.039) 0.05 (0.039) 0.05 (0.039) 


Hedges g effect size for main effects model (95% Confidence Intervals): = +0.06 (-0.14, +0.27). 


'* This was measured using the Progress in International Reading Study (PIRLS) 2011 ‘Students 


Motivated to Read' Scale 
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A small positive effect size was found but this was not statistically significant and so we must conclude 
that there is no evidence of impact on the secondary outcome ‘Students Motivated to Read’. No 
evidence was found for differential effects across participant subsamples (indicated by a lack of 
statistically significant interaction terms). The KS2 pre-test measure was also not statistically 
significant, reflecting its weak correlation with the PIRLS 2011 'Students Motivated to Read' outcome 
(Pearson's r=0.08). 


3.18 Secondary outcome — PIRLS ‘Students Motivated to Read' Scale — dosage analysis 
(Intervention sample only, n=183 across 53 primary schools) 


coef. s.e. coef. s.e. 
Pre-test (KS2 Reading) 0.09 0.064 0.09 0.064 
Dosage Variables 
Days attended (overall) 0.00 0.024 - - 
Days attended (primary school) - - -0.01 0.037 
Days attended (secondary school) - - 0.01 0.030 
Constant 13.60 0.273 13.59 0.275 
-2 Log Likelihood -497.29 -497.22 
School level variance (s.e.) 0.07 (0.911) 0.13 (0.935) 
Pupil level variance (s.e.) 13.36 (1.660) 13.29 (1.659) 
Intra-Class Correlation (s.e.) 0.00 (0.068) 0.01 (0.070) 


The coefficient for attending sessions overall was zero and when separating these out into primary 
and secondary school stages, was close to zero. Therefore we conclude that there was no evidence 
that attendance had an impact on the 'Students Motivated to Read' secondary outcome measure. 
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Cost 


Based on estimates from Unitas, the direct cost that schools would be required to pay for the 
resources and support provided by Unitas for the TextNow Transition Programme is £112 per pupil. 
This assumes a minimum cohort size of 10 and that sufficient schools are in the programme to make it 
viable. In addition, to run the TextNow programme each school must appoint a coordinator to recruit 
and manage volunteer coaches, manage programme delivery and undertake quality assurance and 
reporting. The programme also requires volunteer coach time of twenty minutes per pupil per day for 
75 days, plus time for initial training, preparation for coaching sessions and keeping records. 


The direct costs quoted per pupil are an underestimate of the actual full economic cost of the 
programme as they do not include estimates of the cost of coordinators, coaches and other resources 
in kind. These are important to include as they represent significant ‘opportunity costs’ in that they are 
resources that could be used on other programmes. However, an estimate of the total cost of this 
programme would require a full economic cost-effectiveness analysis and was beyond the scope of 
this present study. 


Schools did not pay for the TextNow Transition Programme and received additional incentives through 
the provision of MyChoice! credits, which young people exchanged for books. Schools were also 
assisted on an individual basis to meet other costs incurred through their participation in this trial such 
as payments for additional teaching assistant time and, where secondary school pupils undertook the 
role of coaches, the cost of transport from the secondary to the primary school. 
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4. Process evaluation 


4.1 Implementation 


Exposure and dosage 


Full dosage for pupils engaged in the TextNow Transition Programme comprised 15 weeks of daily 
one to one 20-minute sessions (five weeks in primary school and ten weeks in secondary school) with 
a volunteer coach. This was supplemented by access to the MyChoice! website so that pupils could 
engage in online activities related to reading and 'buy' books with the credits earned from their 
attendance and participation in the coaching sessions. Pupils were encouraged by coaches to read for 
20 minutes per day in their own time. This aspect of the programme was not formally monitored. As 
detailed in the fidelity section below, there was variation across the participating schools in their 
adherence to guidelines regarding the quantity and content of coaching sessions and in providing 
access to MyChoice! This, in turn, impacted on the extent to which pupils were enabled to access the 
full dosage for the programme. 


Attendance data supplied by the participating schools to Unitas is tabulated in Table 3.13 in the impact 
evaluation section. This shows that the mean number of days on which pupils received coaching in 
primary schools was 17.82 (varying from 0 to 25, against the programme specification of 25). In 
secondary school the mean number of days on which pupils received coaching was 41.99 (varying 
from 0 to 50, against the programme specification of 50). Data on usage of MyChoice! during the 
secondary period of TextNow indicates that some pupils did not receive access to this aspect of the 
intervention. Just under one quarter of pupils in the intervention group (61 of the 252) placed no 
MyChoice! book orders. Most of these (n=41) came from five secondary schools where little or no 
usage of the MyChoice! website was evident and the rest from a further seven schools with low usage. 
Data was not available for usage of MyChoice! in the primary schools. Reasons for variation in 
exposure and dosage are discussed in the fidelity section (4.2). 


Pupils’ engagement, motivation and experiences 


The majority of coaches (84% of primary coaches and 79% of secondary coaches) that responded to 
the coach survey felt that the young people were engaged during the coaching sessions and had not 
encountered any behavioural issues. It is important to note here and elsewhere throughout this section 
where the coach survey is used as evidence, that the response rate was low (primary 11%; secondary 
21%). Although this represents coaches in over 50% of all schools it is likely that responses were 
received from more committed coaches. 


Pupils reported varying levels of engagement and enjoyment of the TextNow programme. Pupils’ 
engagement and enjoyment appeared to be related to their attitude towards reading, as well as being 
greatly influenced by the quality and consistency of the relationship with, and input of, their coach. 
Focus group pupils who reported high levels of engagement and enjoyment also talked highly of the 
relationship they built up with their coaches — who remained the same over the term — and their 
enjoyment of the regular one to one attention in a separate, quiet room. Where the coach relationship 
was good this could overcome a reluctance to read, as one focus group pupil explained: 'Reading’s 
really boring, but | was happy to get out of my lesson to do it because [the coach] was kind and very 
nice’. Pupil engagement was also enhanced where the coaching sessions were perceived to be fun, 
this appeared particularly important for disengaged boys. 


Across the focus groups, pupils who were supported by staff coaches more consistently reported 
greater levels of engagement and enjoyment than those supported by other student coaches. The 
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engagement and enjoyment of pupils supported by other students in their school was dependent on 
the attitude and engagement of their student coach. For example, one pupil reported they read to 
themselves as their coach wasn't listening, and another explained: 'He wouldn't let me read the book | 
wanted to read, he said it was boring! | got fed up with it after that’. Some pupils’ motivation and 
enjoyment were negatively affected by a concern with what they were missing when they attended 
TextNow coaching sessions. In one focus group, school pupils attended TextNow during tutor 
sessions and felt that they were missing out on the fun activities with friends and information-giving 
that was important to them settling in during their first term. Similarly, in another school, where the 
coaching sessions were scheduled during assembly, pupils resented missing their friends and 
explained that there were ‘lots of things we didnt know or find out about because we werent at 
assembly’. While some pupils were happy to come out of lessons, there was also some reluctance 
due to the 'hass/e' of having to catch up and finish class work. It is, however, important to note that 
some pupils did see the benefits of extra reading, even when balanced against missing other school 
activities or a disinterested coach '/ would rather do the reading because it gets you better at doing 
reading". However, a few remained resentful even though they thought their reading had improved. 
While the pupil focus groups provided indicative evidence of the range of activity that pupils missed, 
data on this was not collected across all schools. 


There appeared to be a gender difference in terms of pupils’ engagement with reading for pleasure at 
home or outside of the coaching sessions. Girls across most of the focus groups were generally more 
likely to report that they enjoyed reading in their own time — either on their own or to a member of their 
family. While some of the boys stated that they did enjoy the TextNow sessions, this less frequently 
translated to reading outside sessions ‘The (coaching sessions) were well fun but | still don’t like 
reading on my own - it's boring’: 


Access and use of the MyChoice! website varied across schools. In one of the focus group schools, 
where coaches and pupils regularly recorded points and played online games and activities, this 
seemed to engage and motivate the boys in particular: '/t was awesome, | loved going on it nearly 
every day.’ Whilst most pupils across the other three focus group schools generally enjoyed spending 
their points and ordering books, the recording of these, and use of the website at school, only 
happened irregularly. For student-coached pupils, this required additional input from the TextNow 
school coordinators who they only saw occasionally. Some pupils had problems remembering their 
usernames and passwords which limited their use of MyChoice! at school and at home. Some pupils 
expressed frustration at using the MyChoice! website, as the books they thought were available could 
not be found or ordered when they searched for them, an issue also raised in the coach and 
coordinator surveys. 


School coordinators and coaches — engagement and experiences 


There was variation in the enthusiasm and commitment of schools involved in the TextNow Transition 
Programme. Where there was less commitment from schools, there was less adherence to the 
intended intervention which created fewer or poorer quality opportunities for pupils to engage with the 
programme. 


A number of factors led to lower levels of commitment and, in some cases, school dropout. The most 
frequently mentioned factors identified from a range of evidence sources, spanning feedback from 
Unitas staff; coordinator and coach surveys; informal feedback from secondary school coordinators to 
invigilators and the focus group researcher; and telephone calls to schools to ascertain reasons for 
drop out, were: 


e Lack of senior leader commitment. This tended to occur where recruitment had been via 
academy chains so the ‘buy-in’ of individual heads was not necessarily secured at the start of 
the project. 


e Delays in appointing coordinators and coordinators changing (or, in a very few cases, no-one 
being in place to take on the role). The role of the coordinator was crucial in recruiting and 
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supporting coaches and facilitating access to MyChoice! In addition, secondary coordinators 
played an important role in building relationships with primary schools. 


e Perceived lack of benefit to primary schools. Many primary schools saw no benefit in 
committing to the TextNow programme. SATs had been completed prior to the start of the 
programme, so any improvements in pupils' reading would not improve their published results. 
In addition, maintaining daily reading sessions became difficult because of school trips and 
other planned end of year school activities. Some primary schools also found it hard to release 
staff to attend meetings related to the project. 


e Lack of well-established pre-existing relationships between secondary schools and their 
feeder primary schools. Some secondary schools explicitly used the trial as an opportunity to 
engage their feeder primary schools. However, the quality of the relationship and 
communication between secondary and primary schools varied and, even though Unitas 
helped broker discussions between primary and secondary schools, the intervention was 
easier to implement where secondary schools had stronger relationships with their primary 
counterparts. 


e Duplication of reading initiatives. Some secondary schools already had established reading 
intervention schemes to support the target pupils, so some coordinators were unclear about 
why they were participating in the TextNow programme, or felt forced to participate. 


e Amount of administration/paperwork. Some school coordinators and coaches felt 
overburdened with the administrative requirements of the programme. However, for 
coordinators it was sometimes unclear to what extent the burden was accounted for by trial 
itself rather than programme-related administration. 


Although a ‘Memorandum of Understanding’ (Appendix VI) was supplied to primary and secondary 
schools to help formalise arrangements, Unitas considered that this could have been developed 
further to provide simple but clearer statements of responsibilities. 


4.2 Fidelity 


Training for coordinators and coaches 


Coordinator and coach training programmes (comprising modules, videos and downloadable 
resources) are made available on the Unitas website, and the expectation is that all coordinators and 
coaches will undertake the training and draw on the materials to support them in their role. The vast 
majority of coordinators and coaches in both primary and secondary schools who responded to the 
surveys had engaged at least to some extent with the training resources. Overall, primary coaches 
had higher levels of engagement in the training than secondary coaches. In contrast, there was little 
variation between primary and secondary coordinators' engagement in the training. There was 
stronger engagement by both coaches and coordinators with the online modules than either the 
training videos or downloadable resources (for example, 92% of coaches responding to the coach 
survey reported engaging with the online modules, whereas 75% engaged with the online videos and 
77% accessed the downloadable resources). Reasons given by coordinators who had engaged less 
with the online training materials were: a lack of time, other people having taken on some aspects of 
the role so they did not have to carry out some tasks themselves, or direct contact and emails from 
Unitas had provided sufficient information. Reasons given by coaches for not engaging with, or not 
making more use of, the online training materials were related to their prior knowledge and experience 
and difficulties in accessing online videos. Some more experienced coordinators and coaches 
reported that they did not engage with the guidance as they had already received training in 
supporting reading and found there was too much information for their needs. However, some less 
experienced coordinators and coaches, particularly student coaches, felt that the materials did not 


Education Endowment Foundation 37 


Process evaluation 


provide sufficient guidance. This indicates a need for a more differentiated approach to training to 
maximise adherence to the TextNow programme (as indicated by Coordinator and coach surveys). 


Undertaking the coordinating role 


Approximately 80-90% of coordinators undertook, at least sometimes, most of the key activities and 
tasks required of them as set out in Unitas guidance, with stronger compliance in secondary schools 
than primary schools. Coordinators reported high levels of compliance in both primary and secondary 
schools in meeting with coaches, monitoring pupil attendance, discussing pupil progress and future 
plans, and (in secondary schools only) providing resources, monitoring pupil progress and conducting 
the final review. However, compliance was very low in primary and secondary schools in organising 
trips (for example to a public library), carrying out celebration events or (in secondary schools only) 
sharing progress with parents/carers. In addition, in primary schools at least 25% of coordinators did 
not: (a) motivate coaches or advise coaches on suitable texts; (b) suggest approaches to planning to 
coaches; or (c) advise coaches on the use of MyChoice! with their young person. Coordinators, in the 
relevant survey, cited a lack of time, issues regarding the sharing of tasks with coaches, and (for 
secondary coaches), not knowing the young people when they arrived at the school as reasons for not 
being able to carry out specific tasks. (Coordinator surveys). 


Recruitment and allocation of coaches 


As anticipated in the Unitas guidance, coaches were drawn from a variety of backgrounds. We do not 
have data on the whole coach population. As noted earlier, the low response rate for the coach 
surveys and the likelihood of over-representation of more committed coaches amongst respondents 
means that findings drawn from the coach surveys should be regarded as tentative indications. 
Responses to the coach survey indicate that the majority of coaches in both primary schools and 
secondary schools were either teaching assistants or learning support assistants (primary, 59.5%; 
secondary, 41.7%). Teachers or head teachers were more likely to take on the coach role in primary 
schools (primary, 19.1%; secondary, 5.6%), whereas just over a third of coaches in secondary school 
(36.1%) were peers from the school. Peers were usually drawn from older year groups. Other coaches 
were higher education students, librarians, administrators, and parent link workers. 


Table 4.1 Background of coaches 


n % n % 
Teaching assistant (TA)/ learning support 25 59.5 30 41.7 
assistant 
Teacher or head teacher 8 19.1 4 5.6 
Peer coach from a participating secondary 3 7.1 26 36.1 
school 
Higher education student 0 0 3 4.2 
Other categories with 2 or fewer responses: 6 14.3 9 12.5 
includes librarians; administrators, parent link 
workers and not specified 


The majority of TextNow young people in primaries and half of those in secondary schools worked 
with one coach only. However, around a tenth of all coordinators said that most of their young people 
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in the intervention group in their school had more than one coach each whilst in their school. Coaches 
who shared their coaching with someone else were supposed to ensure effective handover and 
shared planning. However, amongst those coaches who shared their coaching with someone else, 
40% of primary and 21% of secondary coaches did not undertake any shared planning, which may 
have undermined consistency and impinged on young’ peoples’ experiences and 
outcomes.(Coordinator and coach surveys). 


One to one coaching sessions 


As explained above, the exposure, dosage and consistency of the intervention was highly variable in 
both primary and secondary schools, thus undermining the fidelity of the intervention both within and 
across schools. Evidence from the focus groups suggests that many pupils experienced sub-optimal 
levels of the intervention at some stage — whether at primary school, secondary school or both. This 
negatively affected pupil engagement and enjoyment. Nineteen pupils in the intervention group only 
received coaching sessions in their secondary school. 


Coordinators are expected to ensure that coaches know how to make sense of the initial reading 
interview information and the pupils' reading profile, however just under a fifth of primary coaches and 
just under half of secondary coaches were unaware of this requirement. The vast majority of coaches 
who were given the information did take it into account when working with their young people. (Coach 
surveys.) 


As intended, coaches reported the use of a range of strategies for choosing texts and engaging young 
people, but often did not, as set out in the guidance, plan their session. Coaches felt that planning 
would lead to more formality in the sessions, making them less enjoyable for the young people, or said 
that they had not had time to plan. Coaches tended to use ‘summarising texts’, ‘asking questions’, 
‘paired reading’ and 'KWL' techniques to 'a great extent’ rather than the other techniques. Only 50% or 
fewer coaches used ‘visual activities’, kinaesthetic activities’ and 'ICT', with use of visual and 
kinaesthetic activities being lowest in secondary schools. The 20-minute session time constraint also 
led to the coaches using some techniques less often. Coaches also noted that some 
techniques/approaches were not suitable for certain young people. However, a few coaches stated 
that they were not aware that they should be carrying out any of the techniques and approaches. This 
may have been due to not engaging with the training materials. Primary coaches were more likely to 
use specific strategies than their secondary counterparts. The vast majority of sessions took place in 
an area where the coach and young person were rarely disturbed, although in primary schools there 
were some instances where the coaching took place in the classroom while teaching continued, and in 
a few secondary schools coaching took place during a form period lesson. (Coach surveys and pupil 
focus groups.) 


While some pupils experienced 20 minutes a day with a coach, the surveys and pupil focus groups 
indicate that this was not consistently applied. Sessions were reported to have varied from 10 to 30 
minutes in duration and from daily to a few times a week in frequency. For example, in one primary 
school pupils reported that: 'we just did it whenever the teacher had time and came and got us... some 
days we didn't do it because the teacher was busy’; and in another '/ read every day but sometimes for 
only 10 minutes’. The inconsistencies in delivery were even greater at secondary school, particularly 
for pupils with student coaches where the commitment and competence of their student coach was an 
issue. Some pupils coached by other students received little or no consistent structured support, whilst 
other student coaches took the role seriously. In one school with student coaches a pupil noted that: 
‘About seven times no one was there at all so we would just sit there and we just talked to each other’. 
The variation in the approach of student coaches was summed up by a pupil in another school: ‘But 
when you had a good buddy, it was different, much better if they did it properly or we went to the 
library’. Pupils with staff coaches more often reported that they had received more consistent and 
higher quality support than pupils with peer coaches. (Pupil focus groups.) 
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Feedback from school coordinators during pupil focus group visits confirmed that the consistency and 
quality of coaching was impacted by school-related factors including: timetabling issues; some subject 
teachers not allowing pupils out of their lessons or allowing them out on time; staff/student coach 
availability; difficulties in providing cover for coaches when absent and the low commitment of some of 
the student coaches. (Coordinators surveys.) 


Use of MyChoice! 


As noted in the exposure and dosage section, Unitas usage data evidenced considerable variation in 
the extent to which pupils engaged with MyChoice! Almost a third of secondary coaches and a tenth of 
primary coaches who had completed the surveys reported that they had not encouraged the use of the 
MyChoice! website with their pupils. Some coaches commented that they did not know enough, or, in 
a very few cases, anything at all about MyChoice! A few pupils in the focus group who had been given 
their login details at primary school continued to access the website from home even though their 
secondary coach did not use the website with them. Some schools found the MyChoice! site difficult to 
navigate. (Coach surveys and pupil focus groups.) There is insufficient data to ascertain whether the 
lack of use of the MyChoice! website was attributable to pupil choice or access issues. 


4.3 Perceived outcomes 


Coaches and coordinators in both primary and secondary schools reported positive impacts on 
participants, with coaches reporting stronger outcomes than coordinators. Between a third and half of 
coaches and coordinators felt that the TextNow programme had increased the enjoyment and interest 
in reading of all of the young people that they had worked with. Interestingly, they identified a stronger 
impact on the development of participants’ reading skills and comprehension — around 60% of 
coaches and 40% of coordinators felt that the programme had increased young peoples' reading skills 
and comprehension. Positive unexpected outcomes noted by coaches and coordinators were the 
development of relationships between coaches and young people, improved links between primary 
and secondary schools, and the increased confidence and eagerness of the young people involved. 
Some secondary school coordinators that had deployed student coaches noted an increased sense of 
responsibility in the student coaches (Coach and coordinator surveys and feedback from secondary 
coordinators.) 


Negative outcomes were noted by coaches and coordinators in terms of resentment and loss of 
interest from young people due to the time consuming nature of the programme. While focus group 
pupils pointed to the negative impact of missing assemblies, form periods and lessons, there is 
insufficient data to substantiate whether or not this led to negative social or learning outcomes. 


Regardless of the quality and consistency of the intervention, overall most pupils in the focus groups 
felt that their confidence in reading, and their reading level, had improved as a direct result of their 
additional reading, albeit to differing extents. Pupils in three of the four focus group schools reported 
making greater use of their school or public library: '/ borrowed a book and when | was home and 
bored, | started to read it, and | finished it last week... It’s made a difference cos when | was at primary 
school | didn’t like reading, now | wish | could read even more’. Some pupils, mainly girls, reported 
reading more regularly and for longer than before — sometimes reading different authors, or longer and 
more challenging texts: '/ used to read two times a week and now | read every day for about an hour’. 
However, for some pupils, especially boys, their increased confidence and regular in-school practice 
during the TextNow programme did not result in reading for pleasure more frequently. Even at schools 
where the intervention was the most consistent and of the highest quality, a few pupils who did not 
read for pleasure before the intervention still did not at the end of the trial, despite their stated 
enjoyment of the one to one sessions with their coach. 
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Notwithstanding the sometimes negative aspects of their experience, nearly all pupils thought that, on 
balance, the additional reading had benefitted them and they would do it again if they could. The 
factors they felt made the most positive difference were: 


e the one to one attention, particular with a engaged and encouraging coach; 


e having someone to listen to them read and help them concentrate and guide them when they 
got stuck; 


e the regularity of the sessions and the additional practice; 


e the MyChoice! website. 


4.4 Formative findings 


Summary 


There was considerable variation across and within schools regarding adherence to the TextNow 
programme. This was particularly problematic where schools did not feel a strong sense of 
commitment to the programme. Coaching sessions varied in duration and frequency and in the extent 
to which they adhered to the Unitas guidance. Consequently pupils experienced varying levels of 
consistency and quality of coaching. Where the relationship with their coach was positive, respectful, 
and consistent the pupils were most likely to find reading enjoyable. Some of the student coaches who 
were deployed in some schools were uninterested in their coaching role. Use of the MyChoice! 
website also varied across schools. Where pupils had access, many were positive about using the 
site, although some had been unable to 'purchase' the books they wanted as they were not available. 
Coordinator and coach engagement with training was also variable, with the training considered too 
detailed for coaches with a literacy teaching or support background and insufficiently detailed for less 
experienced coaches. 


Despite inconsistency in the implementation of the programme and some pupils' reluctance to miss 
lessons to attend the coaching session, the pupil focus groups indicated that most intervention pupils, 
including the most reluctant readers, thought that their reading had improved as a result of the 
TextNow Transition Programme. Many young people also reported increased confidence, reading 
more, reading for longer periods of time, and reading more challenging books; borrowing books from 
school and public library had also increased. Girls appeared to be keener to read for pleasure than 
boys. Coaches and coordinators considered that the intervention had had a stronger impact on the 
development of young people's reading skills than their enjoyment of reading or their reading levels, 
with coaches reporting stronger outcomes overall than coordinators. 


The process evaluation has highlighted inherent problems in adopting the TextNow programme for 
use at transition. There is little incentive for primary schools to participate and the coaching sessions 
in primary are disrupted due to clashes with end of year activities. In the first term of secondary school 
young people miss out on essential information and friendship-building when they attend coaching 
sessions. 


Necessary conditions for the success of the Unitas TextNow Transition 
Programme 


Our evaluation indicates that successful implementation of the programme requires: 
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e Commitment from primary and secondary schools involved that is underpinned by a sense of 
ownership by the school and a clear understanding of how the intervention will benefit the 
school and pupils. This is crucial to ensuring that the TextNow programme is implemented as 
intended. 


e Full engagement in training by coordinators and coaches so that they are knowledgeable 
about their role and responsibilities, and for those new to supporting literacy development, 
training regarding the use of appropriate strategies. Engagement in training could be improved 
by Unitas developing a differentiated training approach that takes account of coaches’ prior 
knowledge and experience. 


e The selection of coaches that are enthusiastic and committed to supporting young people and 
have the skills to build positive relationships. 


The programme could be improved by resolving the technical difficulties and the clarity of the My 
Choice! website, and ensuring that all books advertised on the site are available. Coaches and 
coordinators recommended that the programme should be less time-consuming and intensive, with 
the duration of the sessions extended but the frequency reduced. They also recommended that only 
one coach should work with each young person, that more preparation time is required, and that face- 
to-face, rather than online, training would have been more effective. 


4.5 Control group activity 


Feedback to Unitas from the schools indicated that three pupils in the control group were given the 
intervention at their primary school. This was faded out at secondary school. No other pupils in the 
control group received coaching sessions or access to MyChoice! for the duration of the project. 
Access to MyChoice! was given to control pupils after the post-test. The small scale of the process 
evaluation did not allow for data gathering from control group pupils so we are unable to make any 
claims about their perceptions of the trial. However, some data does indicate that the intervention 
group pupils resented missing out on activities, such as assemblies and lessons that the control group 
attended. 
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5. Conclusion 


5.1 Key conclusions 


Initially, it should be understood that the trial results and therefore conclusions in this report relate to 
the Unitas TextNow Transition Programme which was designed and implemented (for the first time) 
specifically for this trial. As such, it is important to note that the following conclusions do not refer to 
the established 10-week TextNow programme which targets mid-teens. 


1. The trial has not provided any evidence that the TextNow Transition Programme improved reading 
comprehension or attitudes towards reading for pleasure over the transition from primary to secondary 
school. 


2. On average, pupils who participated in the programme made slightly less progress than similar pupils who 
did not. However, this finding was not statistically significant, meaning that it could have occurred by 
chance. 


3. The programme was found to have a differential effect for pupils eligible for free school meals and their 
peers. A small positive (but not significant) effect was found for pupils eligible for free school meals, while 
a negative (and statistically significant) effect was detected for pupils not eligible for free school meals. It 
is unclear why this differential effect was found. 


4. Higher attendance at the 20-minute daily coaching sessions was found to have a positive impact on 
reading comprehension. However, this was only found to be statistically significant for attendance at 
sessions in secondary schools. Attendance of the coaching sessions was not found to have an impact on 
the secondary outcomes (liking reading and motivation to read). 


5. The programme appeared to be more effective when coaches were highly trained, enthusiastic and 
committed, and when secondary schools worked closely with feeder primaries to coordinate all elements 
of the programme. 


5.2 Limitations 


Despite some issues with recruitment of schools, the final number of secondary schools recruited 
before randomisation (N=34) was above the required number set out in the sample size calculation 
(N=30). Regarding pupil sample size, there were some issues with recruitment and attrition. The initial 
trial size was anticipated to be N=600, however. However, there were some issues with gaining 
consent that led to a lower than anticipated participant sample (N=501 recruited). Furthermore, some 
children were lost to the post-test in both the intervention (N= 53) and control groups (N=56 plus one 
pupil not matched to the NPD ) providing a final analytical sample of N=391 and an attrition rate of 
22%.% Despite the smaller than expected pupil sample size, pre-test comparisons of the intervention 
and control groups indicated that the randomisation had worked well providing two balanced groups 
for evaluation. Therefore, it is suggested that the trial had good internal validity, with no major 
limitations with regard to the study sample size and potential bias between the study groups. 


Conclusion 


With regard to external validity, the trial schools had a slightly lower percentage of pupils with SEN 
statements or on School Action Plus (3.1%) compared to English national secondary state-funded 
school norms. (7.7%); a slightly higher percentage of pupils with English not as a first language 
(16.8%) compared to national norms (13.6%); and a higher number of pupils eligible for freeFree 
school mealsmeal pupils (25.4%) compared to national norms (16.3%). Overall, it is suggested that 
that the schools represented in the study were slightly higher than average on characteristics of 
disadvantage. Furthermore, the drop-out of five secondary schools does not compromise the internal 
validity of the study but does have implications for the external validity. A total of 60 pupils were lost to 
the trial due to the drop- out of the five secondary schools (intervention = 32, control = 28). Schools 
whichwho were less engaged with the programme also tended to drop -out of the testing, meaning 
that the treatment effect is estimated on more engaged schools. 


5.3 Interpretation 


The evaluation results indicated that the Unitas TextNow Transition Programmeprogramme was 
having no significant effects on pupil outcomes i.e., either their reading ability or attitudes towards 
reading for pleasure. However, there was evidence that the programme had a differential impact for 
FSM and non-FSM participants with respect to the reading comprehension primary outcome measure. 
Specifically, the evidence suggests that those eligible for FSM were more likely to experience a small 
positive effect (although this was not statistically significant) and non-FSM participants were more 
likely to experience a small negative effect. In other words, once the research design and sample size 
are taken into account, there is insufficient evidence to conclude that the small positive effect seen 
amongst FSM participants is due to the TextNow Transition Programme. However, amongst non-FSM 
participants the evidence is stronger and it is reasonable to conclude that the negative effect found is 
due to the TextNow Transition Programme experience. 


The process evaluation found that implementing the TextNow programme at transition created a 
number of issues that are not evident in the established 10-week programme for older secondary 
pupils. In primary schools issues included a lack of commitment to a post-SATs initiative that did not 
directly benefit the school and, clashes with trips and end of term activities. In secondary schools, 
pupils resented being removed from assemblies, classes or other activities to attend coaching 
sessions as they missed important information and opportunities to build friendships. Furthermore, the 
most intensive aspects of the intervention took place during school time, with only limited support, via 
MyChoice! during the summer holiday period. These issues indicate that delivering the TextNow would 
require a more- coordinated approach between primary and secondary schools possibly with added 
support during the summer break for schools and pupils. 


Fidelity problems were most marked in schools that did not feel a strong sense of commitment to the 
programme. As a consequence, some pupils did not receive the intended amount or frequency of 
coaching and/or received no, or limited, access to the MyChoice! website. In some instances where 
student coaches were deployed, they lacked commitment to supporting their pupil and the quality of 
coaching was perceived to be poor by the recipient pupil. The process evaluation findings indicate that 
for transition programmes to be successful, leaders and all staff involved in secondary schools and 
their feeder primary schools need to committed and engaged and feel a sense of ownership of the 
programme. In addition, effective communication between primary and secondary schools is essential, 
as is the high quality coordination of the programme and ensuring that everyone engaged in managing 
the programme and supporting young people undertake high quality training that takes account of their 
prior knowledge and experience. 


There are some salient points in the literature that can help with the interpretation of the results of this 
evaluation. A systematic review of RCT evaluations of programmes that deploy volunteer tutors 
highlights the importance of the programmes being well structured in order to produce positive results 
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(Ritter et al., 2011). In this review a high degree of programme structure was typified by three 
elements, namely: a focus on learning specific literacy skills; regularly timetabled sessions; and the 
deployment of well-trained tutors. Reflecting on these three key factors of tutoring programme 
structure, and in light of the current results, it can be said firstly that the TextNow Transition 
Programme was timetabled but there were problems in fidelity to this timetabling (particularly in the 
primary schools). Secondly, the focus of the sessions was fairly flexible both in choice of text and 
literacy instruction provided. Furthermore, from the process evaluation, building and maintaining 
relationships was a large part of the focus in the sessions rather than learning specific literacy skills. 
Thirdly, there were problems with fidelity relating to the participation in training by the coordinators and 
coaches, and coaches with no prior experience of supporting literacy initiatives were given insufficient 
training in how to develop specific literacy skills. 


A number of these implementation problems stem from contextual issues. Firstly, the fact that the 
programme was running in the transition period, and secondly that it was provided for primary school 
and young secondary students. A high level of coordination between two schools (i.e. the secondary 
school and a feeder primary school) would be required to provide consistent highly-structured 
sessions that focus on literacy skill development. However, McGee et al. (2004) suggest that good 
links between secondary schools and their feeder primary schools are rare, and coordination 
regarding individual pupil needs and past achievements are often neglected. In addition, previous 
evaluations of the related TextNow programme for pupils predominately in their mid-teens showed that 
age was a significant predictor of effectiveness with the greatest benefits of this programme being 
provided to older teens (Brooks, Tarling & Adams, 2011)" The reading age test used as a measure of 
effectiveness in Brook, Tarling & Adam's (2011) evaluations was able to capture and these contextual 
issues may contribute to explaining the lack of positive results on the primary outcome measure. 
However, as the earlier discussion has indicated, it is possible that this may be due to the lack of focus 
on key literacy skill development and the need for a strong supporting structure to underpin skill 
development. 


At this point, it is worth exploring the notion of book choice as a central tenet of the TextNow 
programme. Indeed there is growing practice literature to support the link between choice and 
motivation to read (Flowerday & Schraw, 2000; Gambrell, 1996; Hunt, 1970; Sanacore, 1999; Worthy, 
Moorman, & Turner, 1999). However, there is little systematic evidence of causality or theory of 
change to explain the relationship between these two factors (Schraw, Flowerday, & Reisetter, 1998; 
Flowerday, Schraw, & Stevens, 2004). Conversely, there is systematic evidence indicating that 
tutoring programmes need to be highly structured to be effective (Ritter et al., 2011). Further work is 
therefore needed to explore the theories of literacy change underpinning these potentially 
contradictory approaches. An explicit logic model for the TextNow Transition Programme, or any 
similar programme, could help clarify the short-, medium- and long-term outcomes as well as any 
intended causality or underlying theory of literacy change. 


5.4 Future research and publications 


As indicated there is a growing practice literature on the impact of pupil book choice on student 
motivation to read. However, there is a lack of rigorous and systematic literature in relation to the 
effects of book choice. Therefore, one area for future research is to look at the effectiveness of book 


The reading age test used by Brooks, Tarling and Adams (2011) measured reading progress for 
children and young people up to the chronological age of 16 years 6 months, who had a reading age 
of 16 years or less. 
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choice on motivation to read and how this interacts with different types of literacy interventions such as 
tutoring programmes. Further research is also need to explore the reasons for the differential effect of 
book-choice programmes on FSM and non- FSM pupils 


The evaluation team will be seeking to publish the trial findings in academic journals. 
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Appendices 


Appendix! Outcomes 


Table Al.1 KS2 Reading Fine Score (Pre-test) 


Mean (sd) 24.3 (4.27) 24.7 (4.29) 24.5 (4.28) 
Median (IQR) 25.1 (6.2) 24.4 (5.0) 25.1 (5.8) 
Max 33.2 34.0 34.0 

Min 15 15 15 

n= 192 199 391 


Table Al.2 GL New Group Reading Test statistical summary 


Mean (sd) 88.9 (10.88) 88.8 (11.55) 88.8 (11.21) 
Median (IQR) 89.0 (15.8) 88.0 (16.0) 89.0 (16.0) 
Max 129 118 129 

Min 69 69 69 

n= 192 199 391 


Deriving the secondary PIRLS items 


PIRLS ‘Students Like Reading’ scale 


This scale was constructed using eight questionnaire items using the guidance provided by Martin et al., (2012). 
These eight items (or components) are shown in the table below along with the responses from the TextNow 


transition programme RCT sample. 


Table Al.3 PIRLS 'Students Like Reading’ components 


| like talking about what | read with other 78 (21%) 83 (22%) 117 (82%) 93 (25%) 371 
people 
| would be happy if someone gave me a_ 50 (14%) 66 (18%) 127 (35%)  123(34%) 366 
book as a present 
| would like to have more time for reading 59 (16%) 88 (24%) 139 (88%) 79 (22%) 365 
| enjoy reading 45 (12%) 55 (15%) 120 (82%)  152(41%) 372 
Agree a Agree a Disagreea Disagreea n= 
lot little little lot 
| only read if | have to 89 (24%) 123 (83%) 87 (23%) 75 (20%) 374 
| think reading is boring 51 (14%) 71 (19%) 101 (28%)  143(39%) 366 
Never Once / Once / Everyday n= 
twice a twice a 
month week 
| read for fun 51 (14%) 50 (13%) 159 (42%) 115(31%) 375 
| read things that | choose myself 22 (6%) 60 (16%) 145 (89%)  146(39%) 373 


The reliability of using these eight components to create the 'Students Like Reading' scale was assessed using 
the Cronbach alpha statistic. For the sample of participants involved in the TextNow transition programme RCT 
evaluation, Aloha was calculated to be 0.86—slightly higher than that provided for the England sample in the 
PIRLS documentation (Alpha=0.85) suggesting a good degree of internal consistency. The table below provides 
a statistical summary of the PIRLS 'Students Like Reading’ scale. 


Table Al.4 PIRLS ‘Students Like Reading’ scale statistical summary 


Mean (sd) 14.4 (5.52) 14.6 (5.95) 14.5 (5.74) 
Median (IQR) 15.0 (8.0) 16.0 (8.0) 15.0 (8.0) 
Max 24.0 24.0 24.0 


Min 0.0 0.0 0.0 


n= 


158 


173 


331 


PIRLS ‘Students Motivated to Read' scale 


This scale was constructed using six questionnaire items using the guidance provided by Martin et al., (2012). 
These are shown in the table below along with the responses from the TextNow transition programme RCT 


sample. 


Table Al.5 PIRLS 'Students Motivated to Read' components 


| like to read things that make me think 46 (12% 63 (17%) 152 (40%) 118(31%) 379 
It is important to be a good reader 16 (4%) 32 (9%) 97 (26%) 230 (61%) 375 
My parents like it when | read 17 (5%) 29 (8%) 105 (28%) 222 (60%) 373 
| learn a lot from reading 23 (6%) 52 (14%) 132 (35%)  168(45%) 375 
| need to read well for my future 15 (4%) 39 (10%) 116 (31%) 209(55%) 379 
| like it when a book helps me imagine | 37 (10% 44 (12%) 122 (33%)  162(44%) 365 
other worlds 


For the sample of participants involved in the TextNow transition programme RCT evaluation, Alpha was 
calculated to be 0.8 for the PIRLS ‘Students Motivated to Read' scale which was slightly higher than that 


provided for the England sample in the PIRLS documentation (Alpha=0.78). 


The table below provides a statistical summary of the PIRLS 'Students Motivated to Read' scale. 


Table A1.6 PIRLS ‘Students Motivated to Read’ scale statistical summary 


Mean (sd) 13.3 (3.80) 13.6 (3.69) 13.5 (3.75) 
Median (IQR) 14.0 (5.0) 15.0 (4.0) 14.0 (4.0) 
Max 18 18 18 

Min 0 0 0 

n= 169 183 352 
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Table Al.7 School level: outcomes across the 53 PRIMARY schools 


PAN 0) of=yare | (exer) 


Mean (sd) 24.49 (2.565) 88.63 (5.869) 14.65 (2.572) 13.35 (2.048) 
Min 18.77 73.00 9.50 9.00 
Max 30.12 106.00 21.00 17.50 


Table Al.8 School level: outcomes across 29 SECONDARY schools 


Mean (sd) 24.72 (2.302) 88.48 (4.448) 14.80 (2.308) 13.43 (1.591) 
Min 19.02 78.00 11.05 9.00 
Max 29.56 97.69 21.00 17.50 
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Appendix Il Approach to modelling and terms used in 
tables 


All.1 Approach to modelling 


The models were fitted in stages: 


First, a main effects model that simply fits the coefficient terms for all explanatory variables. This 
assumes that no interactions between the terms are present. To explore whether this is a reasonable 
assumption, a series of models were constructed that fitted specific interactions with the programme 
under evaluation. In doing the models are exploring whether there is evidence for the programme 
having a differential (rather than consistent) impact for differing groups of participants: 


Pre-test interaction: These models explore whether the impact of the programme under evaluation 
depends upon prior attainment (at KS2). For example, the programme might have a greater impact 
amongst higher attainers (indicated by a positive coefficient on the interaction term) or it might have a 
greater impact amongst lower attainers (negative coefficient on the interaction term). 


Gender interaction: These models explore whether the impact of the programme under evaluation 
depends upon gender. For example, the programme might have a greater impact for females 
(indicated by a positive coefficient on the interaction term) or it might have a greater impact amongst 
males (negative coefficient on the interaction term). 


FSM interaction: These models explore whether the impact of the programme under evaluation 
depends upon FSM status. For example, the programme might have a greater impact for FSM 
participants (indicated by a positive coefficient on the interaction term) or it might have a greater 
impact amongst non-FSM participants (negative coefficient on the interaction term). 


Effect Size: This is a standardised measure that is calculated from the model coefficient. 
Hedges g is the effect size measure used (see Appendix Ill for more detail on calculating this). The 
measure standardises so that units are converted into standard deviations - which, unlike the raw 
coefficient, can be directly compared across many models (i.e. primary and secondary outcome 
measures). Higher values indicate greater statistical impact. 


All.2 Key terms used in the modelling tables 


Coef: This is the estimated coefficient term for each of the explanatory variables. This term 
can be positive, zero or negative. A positive coefficient indicates a higher score on the outcome 
variable. For example, if a positive coefficient is seen for the gender variable, this tells us that when 
taking other explanatory variables included in the model into account, on average females got higher 
scores on the outcome variable. Similarly, a negative coefficient indicates a lower score on the 
outcome variable compared with their male peers. For example, if a negative coefficient is seen for the 
FSM variable, this tells us that when taking other explanatory variables included in the model into 
account, on average FSM participants got lower scores on the outcome variable compared with their 
non-FSM peers. A zero coefficient indicates that there is no differences in the outcome variable. For 
example, if a zero coefficient is seen for the pre-test variable, this tells us that scores on the outcome 
variable are not associated with pre-test scores. 


Coefficients are shown in the units of the outcome variable which means that comparing coefficients of 
one explanatory variable across the primary and secondary outcomes is problematic (because of the 
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differing units). This is one reason why effect sizes are used to help interpret since they standardise 
the units and enable direct comparisons to be made. 


Standard Error (s.e.) — This is the standard error for each coefficient term. These are used to give an 
indication on how precise the estimated coefficient term is (smaller values indicate greater precision). 
The standard error takes account of sample size and are widely used in tests of statistical significance 
and calculation of confidence intervals. 


Constant This is the average (mean) of the outcome variable once all of the explanatory 
variables have been included. This represents the mean score for the reference group of the model - 
i.e. non-FSM males who attained a mean score on their KS2 reading test. To calculate the mean for a 
different group, a coefficient term would need to be added - e.g. for non-FSM females who attained a 
mean KS2 attainment score, the gender coefficient would need to be added. 


-2 Log Likelihood This is the total amount of variation remaining once each model has been 
fitted and is conceptually similar to the Residual or Error Sum of Squares used within linear regression 
and ANOVA type analyses. 


School level variance: This is the amount of school-level variation that remains once the model is 
fitted. A standard error term is included with this estimate. 


Pupil level variance: This is the amount of pupil-level variation that remains once the 
model is fitted. A standard error term is included with this estimate. 


Intra-Cluster Correlation (ICC) - This is the proportion of variation that is found at the higher level of 
the model - i.e. the school level. It is usual for this proportion to be much smaller than that found at 
the individual pupil level. 
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Appendix Ill Effect size 


An effect size is a statistical estimate of the strength of a phenomenon in standardised units. In the 
context of this research, the effect size provides an indication of the difference between the 
intervention and control groups for the three outcome measures. Whilst the model coefficients also 
provide an indication of this, the effect size standardises these coefficients so that they can be 
compared directly with each other and across other research studies. Without standardisation, the 
size of coefficient is dependent on the scale and units of the outcome measure and so it is not 
possible to compare these directly. 


As specified by the EEF, the effect size calculated in this report is Hedges g. This is a similar effect 
size measure as Cohens d but uses a standard deviation that is pooled between the intervention and 
control groups. The tables that follow provide some summary detail on the three outcome measures 
and how Hedge's g was calculated. 


Whilst we have used the post-test standard deviations in the calculations of effect size in this report, ® 
we provide detail on the pre-test measure to enable readers to use this detail to re-calculate them. 


Pre-test measure (KS2 Reading fine score) 


Three tables are shown. The first relates to the main analyses that involve all participants. The 
second and third are for the FSM and non-FSM subsamples respectively and relate to the subsample 
analyses for the primary outcome (GL New Group Reading Test). 


Table Alll.1 Main analyses: KS2 Reading Score—all participants 


Mean 24.3 24.7 24.5 
Standard deviation 4.27 4.29 4.28 
n= 192 199 391 
Pooled standard deviation 4.28 


Mean 24.0 24.1 24.1 


Standard deviation 3.90 4.11 3.98 


'© This is so that the approach is consistent to the one used in the EEF Booktrust Summer Active Reading RCT evaluation 
report, which was also a literacy transition trial related to reading for pleasure. 
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57 


116 
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Pooled standard deviation 


4.00 


Table Alll.3 Subsample analyses: KS2 Reading Score—non-FSM participants only 


Mean 24.5 24.9 24.7 
Standard deviation 4.43 4.35 4.39 
n= 133 142 275 
Pooled standard deviation 4.39 


Primary Outcome (GL New Group Reading Test) 


The standard deviations shown in these three tables are used to calculate the hedges g statistics in 
this report. The first relates to the main analyses that involve all participants. The second and third 
are for the FSM and non-FSM subsamples respectively and relate to the subsample analyses for the 


primary outcome. 


Table Alll.4 Main analyses: primary outcome measure—all participants 


Mean 88.9 88.8 88.8 
Standard deviation 10.88 11.55 11.21 
n= 192 199 391 
Pooled standard deviation” 11.23 


Table Alll.4 Subsample analyses: primary outcome measure—FSM participants only 


Mean 


Standard deviation 
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n= 59 57 116 


Pooled standard deviation 11.30 


Table Alll.5 Subsample analyses: primary outcome measure—non-FSM participants only 


Mean 90.2 89.1 89.6 
Standard deviation 10.54 11.61 11.10 
n= 133 142 275 
Pooled standard deviation 11.11 


Table Alll.6 Secondary outcome 1 (PIRLS 2011 'Students Like Reading’ scale) 
(Just one table is shown for both secondary outcomes because no subsample analyses were conducted.) 


Mean 14.4 14.6 14.5 
Standard deviation 5.52 5.95 5.74 
n= 158 173 331 
Pooled standard deviation 5.75 


As shown in Appendix |, this is a pre-validated scale that is a composite of eight questionnaire items. 
A list-wise missing values approach was adopted which led to a compounded proportion of 15% 
(n=60) of missing responses on this secondary outcome. 


Table Alll.7 Secondary outcome 2 (PIRLS 2011 'Students Motivated to Read’ scale) 


Mean 13.3 13.6 13.5 
Standard deviation 3.80 3.69 3.15 
n= 169 183 352 
Pooled standard deviation 3.75 
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As shown in Appendix I, this is a pre-validated scale that is a composite of six questionnaire items. A 
list-wise missing values approach was adopted which led to a compounded proportion of 10% (n=39) 
of missing responses on this secondary outcome. 


Effect Size Worked Example 


Referring to Table 3.10 (Section 3.5, Outcomes and Analysis), the main effects model shows a 
coefficient of -0.72 for the intervention group with a standard error of 0.874. 


The standard error can be used to calculate 95% confidence intervals for the coefficient: 
95% Confidence Interval = Coefficient +/- (1.96 x standard error) = -0.72 +/- (1.96 x 0.874) 
Upper limit of confidence interval = -0.72 + 1.713 = + 0.988 

Lower limit of confidence interval = -0.72 - 1.713 = -2.436 


This tells us that, once random variation is taken into account, the coefficient cannot be confidently 
distinguished from zero. 


The coefficients and confidence intervals can be converted into an effect size by dividing them by the 
standard deviation. For Cohen's d, this would be the standard deviation of the primary outcome for all 
participants (i.e. s=11.21) whilst for hedges g, the standard deviation would be the one that is pooled 
from the control and intervention group (s=11.23). 


The pooled standard deviation is calculated using the following formula: 


= {@ — 1)s? + (no — 1)82 


ny+tng—2 


For the primary outcome, this is calculated to be s= 11.23 


Converting the coefficient and 95% confidence interval limits into (Hedge's g) effect sizes: 


Hedges g = coefficient / pooled standard deviation = -0.72/ 11.23 = - 0.06 
Hedges g confidence interval upper limit = +0.988 / 11.23 = + 0.09 
Hedges g confidence interval lower limit = - 2.483 / 11.23 =-0.22 


As reported in the 'Outcomes and Analysis' section: 


‘Hedges g effect size for main effects model (95% Confidence Intervals): = -0.06 (-0.22, +0.09)' 
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Appendix IV Measuring MyChoice! Engagement 


Details on the number of pupils per secondary school who placed an order for books on the MyChoice 
website were provided by Unitas. The proportion of pupils who placed an order ranged from 0% (in 4 
schools) up to 100% (15 schools) with an average of 72% of pupils placing an order per school. 


Pupils placing 
book orders 


n= 29 
Mean (sd) 72% (37 pp) 
Min 0% 


Max 100% 
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Appendix V-_  Parent/ carer consent form 


oe 
MD) Queens University Sheffield | Centre for 
VY Belfast Hallam _| Education and 


>CreateAchieve University | Inclusion Research 


TextNow Transition Programme 
Parent/carer information sheet/ consent form 


Your child has been chosen to take part in a trial of the TextNow programme which has been designed to 
develop pupils’ reading skills during the transition from primary to secondary school. 


The Centre for Education and Inclusion Research (CEIR) at Sheffield Hallam University and the Centre for 
Effective Education at Queen's University Belfast (‘the evaluators’) have been commissioned by the Education 
Endowment Foundation to carry out a randomised control trial evaluation of this programme. In a randomised 
control trial children are allocated to either an intervention group or a control group. The intervention group 
take part in the programme being tested (TextNow), whereas the control group do not receive any additional 
benefits until the trial has finished. 


This information sheet is to help you decide if you want to give consent for your child to be included. Once you 
have read the information and have discussed this with your child please complete the tear off slip at the end 
and return it to your child's teacher. 


What is the TextNow programme? 


The TextNow programme provides structured one to one reading sessions for 20 minutes each weekday for a 
total of fifteen weeks (five weeks at the end of year 6 and 10 weeks at the beginning of year 7), delivered by 
reading coaches recruited by the schools. Through their participation in these coaching sessions children can 
earn credits with which to buy their own books. 


Children in the intervention group take part in all elements of the TextNow programme. Children in the control 
group do not take part but will be given vouchers to buy books after the TextNow coaching sessions 
evaluation has finished. 


What does the evaluation involve for all children? 


We will be collecting teacher assessment data on all children in the intervention and control groups from your 
school. This will be matched with data from the National Pupil Database such as gender and key stage results. 


A researcher will be visiting your child's secondary school to observe the administration of a reading test at the 
end of the autumn term. The results will be matched to the data previously collected. The data will be used to 
examine the effectiveness of the TextNow programme. 


What else does the evaluation involve for children in the intervention group? 


Children in the intervention group will also be asked to complete a short online survey and/or take part in a 
focus group during the school day at the end of the TextNow programme. 
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There will also be other assessments of reading levels, attitudes to reading and confidence in reading at 
various points throughout the trial. 


What will happen to the evaluation data? 


All pupil data, including test results, will be held confidentially and in compliance with the Data Protection Act. 
Participating schools and children will not be named in any reporting. Anonymised data will be shared between 
Unitas, the evaluators and the Educational Endowment Foundation for current and future research purposes. 


If during the programme you no longer wish your child to take part in the trial please contact the evaluation 
project manager (contact details below). 


If you have any further queries please contact us using the details below. 


Ella Bourne Lucy Clague 
TextNow Programme Manager Evaluation Project Manager 
Unitas Centre for Education and Inclusion Research 
Ella.bourne@unitas.uk.net Sheffield Hallam University 
07557 767721 |.clague@shu.ac.uk 
0114 225 6066 


Please cut off and return this consent slip to your child's teacher. Keep the letter for 
your information. 


TextNow Transition Programme 


| have read the information sheet and understand what is involved for my child in participating in this trial and how data will be 
used and stored. 


| give permission for my child to take part in the TextNow evaluation and if selected for the intervention group to attend 
coaching sessions to develop their reading skills. 


(Ge a) | fe al as = Bae ie eo a ee 


SIGNe! 5.1. s.dait A doles eet et ee Ped eR od atta hth aks pdac at tented tudes Parent/carer 
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Appendix VI Memoranda of Understanding for Primary and 


Secondary Schools © 
UNITas 


Primary Schools Achieve 


The Hub 
Candid Arts 

5 Torrens Street 
London 

EC1V 1NQ 


To: [Insert name and address of Licensee. Example: 


: ? contact@unitas.uk.net 
Name of school (‘the Licensee”)"” @ 


Date: [/nsert date] 
Dear [insert name of authorised representative of the licensee e.g. the Head Teacher] 
Licence 


This letter is to confirm Unitas’ agreement to permit the Licensee (details of which are set 
out above) to use the intellectual property and materials listed in Schedule 1 (“the 
Property’) in connection with the TextNow programme (“the Programme’) for young 
people, as it is used in the TextNow Transition Pilot (“the Pilot’), funded by the Education 
Endowment Foundation. 


Any references to “you” in this letter are references to the Licensee organisation. 
1. Licence 


1.1 Unitas appoints the Licensee as its non-exclusive licensee to use the Property, 
including the registered and unregistered trade marks of Unitas listed as part of the 
Property in Schedule 1 (“the Marks’) in [insert the establishment in which the 
Licensee will use the Property]|'® (“the Site”). The licence lasts for the term of this 
agreement from 1* April 2013 until 1st September 2013, or until terminated under 
paragraph 4 below or extended in accordance with paragraph 6.1 below. 


Unitas may as it chooses from time to time provide web or telephone support to the 
Licensee relating to the Property and the Programme. 


” The Licensee is the organisation to which the Property is to be licensed to under the MyChoice! 
Reward Scheme. 

'8 The establishment will be the geographic area in which the particular TextNow programme service 
is to be provided by the Licensee 
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2.1 


3.1 


3.2 


3.3 


3.4 


3.5 


3.6 


The Property may be used with up to [insert number] of people who are in Year 6 
during the licence period (“young people”), who must complete the Programme by 
the end of the summer term 2013. This pilot cohort of young people may be 
exceeded only with the written consent of Unitas. 


If the Licensee wishes to use the Property outside of the Site or to allow another 
organisation to use the Property it may only do so with Unitas’ written consent, not to 
be unreasonably withheld. If Unitas provides its consent for the Licensee to use the 
Property outside of the Site and/or with other organisation(s), the Licensee agrees 
that it remains wholly responsible for its obligations under the Programme and this 
agreement. However, Unitas may enter into a separate arrangement or agreement 
with those other organisation(s). 


Fee 
There will be no fee payable by the Licensee 
[Insert name of licensee]’s obligations 


The Licensee acknowledges that Unitas is the owner of all rights in and to the 
Property, and agrees: 


to use the Property only for the purposes outlined in Schedule 2; 


to comply at all times with the TextNow Programme Specification at Schedule 3 (“the 
Programme Specification”) and any guidelines Unitas may provide from time to 
time; 


not to bring Unitas or any of Unitas’ funders or partners, or the Property into disrepute 
in any way; 


to obtain Unitas’ prior written approval to all materials (in any media), which bear the 
Marks and to display Marks of funders of the Programme on such materials if 
requested to do so by Unitas; 


to maintain the distinctiveness, reputation and value of the Property and to promptly 
cease any use of the Property that in Unitas’ opinion is not consistent with this at 
Unitas’ request; 


to comply with any Programme evaluation process in accordance with the 
Programme Specification and Unitas’ reasonable instructions issued from time to 
time, including but not limited to: 


3.6.1 the prompt facilitation of any visits to the Site by Unitas and/or the evaluation 
team appointed by the Education Endowment Foundation (the Evaluators) 
legislation, guidance and regulations at all times); 


3.6.2 provision of pupil data to Unitas and the Evaluators (Including but not limited 


to Unique pupil identifier number and National Curriculum teacher assessment 
with a separate list matching pupil name to pupil identifier number) 
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3.7 


3.8 


Ad 


4.2 


3.6.3 completion of a pre Programme assessment of each young person 
participating in the Programme; 


3.6.4 completion of a weekly register during the Programme that will be emailed to 
Unitas no later than 12 midday on the Monday of the following week; 


3.6.5 the participation of the coordinator and coaches in Programme evaluation 
through completion of profiles, online surveys, and interviews, the responses 
to which the Licensee agrees may be used by Unitas and the Evaluators for 
its evaluation, promotional and marketing purposes; 


3.6.6 the participation of young people in completion of surveys and other 
evaluation activities; 


3.6.7 the participation of coaches and young people in the MyChoice! book 
selection and purchase system. Coaches will encourage and facilitate the 
young people to select books on MyChoice! and use the credits they have 
earned to purchase books of their choice. However Unitas bears no 
responsibility for the books provided on MyChoice! and it is the responsibility 
of the coaches and coordinator to monitor and approve book choices 


to ensure that all coaches have undertaken any TextNow online training provided by 
Unitas from time to time; 


subject to paragraph 5.3 below, to comply with all legislation and statutory duties of 
the Licensee insofar as they relate to the Programme, including but not limited to: 


3.8.1 all applicable health and safety legislation; 
3.8.2 all applicable safeguarding legislation; 
3.8.3 all applicable equal opportunities legislation; and 


3.8.4 the Data Protection Act 1998 and all other applicable data protection 
legislation. 


Termination of Licence 


Unitas may terminate this agreement (including the licence) at any time on four 
weeks’ written notice to the Licensee and also reserves the right to terminate it 
immediately if:- 


4.1.1 the Licensee does anything which in Unitas’ opinion brings, or is reasonably 
likely to bring, the Property or reputation of Unitas into disrepute or is inimical 
to the charitable objects of Unitas; and/or 


4.1.2 the Licensee repeatedly breaches any of the terms of this agreement, or 
breaches any material term of this agreement. 


On termination of this agreement, the licence granted under this agreement shall 
cease immediately, meaning the Licensee’s right to use the Property shall cease 


immediately. The Licensee shall not after termination use any materials or intellectual 
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5.1 


a2 


5.3 


6.1 


6.2 


6.3 


6.4 


6.5 


property which are or is similar to the Property or might be likely to cause confusion, 
association with or damage to Unitas. 


Limitation of Unitas’ liability 


The parties acknowledge that the Licensee is solely liable for any loss, cost or 
damage relating to this agreement or the Programme and that Unitas shall not be 
liable for any loss, cost or damage suffered relating to or as a result of the Licensee’s 
participation in the Programme or either party's compliance with this agreement 
(except in the case of death or personal injury caused by Unitas’ negligence in which 
case no limitation applies). Nothing in this paragraph limits or excludes any liability for 
fraud. 


The Licensee agrees to reimburse Unitas in respect of any losses, expenses or 
damage Unitas may incur (including direct, indirect, special and consequential loss, 
and any loss of reputation or loss of profits, including reasonable legal costs and 
disbursements) as a result of any breach by the Licensee of any term of this 
agreement or as a result of any of its actions or omissions, relating to Unitas and/or 
the Property. 


It is the Licensee’s sole responsibility to ensure the Licensee complies with its 
statutory duties (including but not limited to those listed at paragraph 3.8 above) and 
Unitas shall not be responsible in any way for the Licensee’s failure to comply with 
such duties. 


General 


Subject to any amendments or variations to the Property in accordance with 
Schedule 1 or permissions granted under paragraph 1 above, this agreement 
constitutes the entire agreement between Unitas and the Licensee and shall not be 
varied or amended except by written agreement between the parties. 


The Licensee agrees that all data provided to Unitas may be shared with the 
Evaluators. 


The Licensee shall not assign or sub-licence any of its rights under this agreement 
without the prior written consent of Unitas. 


This agreement shall not constitute the parties as partners nor is the Licensee entitled 
to act as nor represent itself as agent for Unitas, nor to pledge Unitas’ credit. 


This agreement is governed by and shall be construed in accordance with the laws of 
England and Wales and the parties irrevocably submit to the exclusive jurisdiction of 
the courts of England and Wales. 


If you have any queries or concerns about agreeing to these terms, please feel free to call 
Ella Bourne on 07557 767721. Otherwise, please confirm the terms are agreed by signing, 
dating and returning to Unitas the enclosed copy of this letter by post, fax, or email showing a 
scanned signature. 


With many thanks. 
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Yours sincerely, 
Signed for and on behalf of Unitas 
Name: Martin Stephenson 


Position: Chief Executive Officer 


my SAL 


Signed for and on behalf of [name of Licensee]"° 


Date: 


Name: 


Position: 


Date: 


'? This agreement must be signed by an authorised signatory only 
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Schedule 1 
The Property 


Unitas shall provide the following material in such form and in such volume as it shall 
in its sole discretion decide from time to time. The Property may be changed or varied 
from time to time on written notification to the Licensee by Unitas. 


The materials 


e Credits to enable the coordinator to obtain a starter library of books for use on the 
TextNow programme 

e Access to online training for TextNow coaches and coordinators including 
downloadable resources. 

e Group based training resources 

e Electronic record sheets for pupil profile, testing and attendance data 

e Access to MyChoice!, an online awards scheme, and up to 1250 credits(points) per 
young person (credits to be generated according to attendance data from coaching 
sessions, as submitted through the electronic record sheets) 

e For this Transition Pilot an additional allocation of MyChoice! credits will also be 
made available to each young person in the trial group at the beginning of the pilot, to 
enable them to choose books for summer holiday reading. 


(Young people participating in the Control group of the pilot will receive a ‘Thank you’ 
award of vouchers to buy books at the end of the pilot, in Year 7.) 


The Marks 

A ‘TextNow’ 
UK trade mark — number 2576568. 
Status: registered. 

2. ‘Unitas’ 


UK trade mark — number: 2515758. 
Status: registered. 


Schedule 2 
The Licensee shall use the Property: 


e in accordance with the agreement; 
e in accordance with the requirements of the pilot 
e inline with Unitas’ instructions from time to time; 
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or educational purposes only; and 
in accordance with the Programme Specification at Schedule 3. 
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Schedule 3 

TextNow Programme Specification 

(for Primary schools taking part in the TextNow Transition Pilot) 
1. Introduction 


The TextNow transition pilot is a randomised controlled trial evaluation of the ‘Unitas: 
TextNow Transition Programme’ for improving reading outcomes among young people in 
transition between Primary and Secondary school 


Unitas will deliver their TextNow Transition Programme to 300 of the 600 participating pupils 
between May and December 2013 (the trial group). The remaining 300 young people will 
receive no intervention during the time period (the control group). 


Pupils in both groups will be assessed at various points throughout the trial to provide 
comparative data for evaluation. It is therefore extremely important that pilot schools maintain 
the integrity of the study by ensuring that control group pupils complete the assessments 
according to the pilot schedule and that they do not receive any alternative intervention 
during the period of the trial. 


This document outlines the TextNow programme for young people who have been identified 
as participating in the trial group of the TextNow transition pilot and who are likely to benefit 
from increasing their motivation to read and their enjoyment of, and proficiency in, reading. 


It is designed to enable organisations delivering TextNow to gain a clear understanding of: 


e what is involved in delivering TextNow 
e the intended benefits 
e the obligations on the part of the host organisation. 


TextNow aims to help young people to be more engaged with reading and to read more 
fluently. This can contribute towards preventing social disadvantage and social exclusion and 
enable young people to more fully engage with their education and training. TextNow was 
developed to work with the target group of young people aged 10-19 years old. For the 
purposes of the TextNow transition pilot the selected pupils will be from Year 6 moving into 
Year 7. 


The TextNow transition pilot involves trained reading coaches working with individual young 
people in Year 6 for daily 20 minutes sessions over five weeks, (followed by a similar ten 
week programme at the beginning of Year 7). A project coordinator, designated by the school 
manages the delivery of the programme at each site. 
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2. Programme Structure 
The objectives of the TextNow programme are to: 
1. increase young people’s reading. 
2. enhance young people’s enjoyment of reading. 
3. improve the reading skills of participating young people, including their 
understanding of what they read. 
Each school will: 


e Ensure that informed parent/carer consent is obtained for each young person 
selected to participate in the pilot, for both control and trial groups. 

e complete all the appropriate records and assessments for every participating young 
person (trial and control group) and return them regularly to the Unitas team where 
required 

e participate fully in the evaluation process, through completion of surveys and 
interviews 

e establish close links with parents/carers to ensure that all those involved are working 
towards the same goals 

e acknowledge Unitas in all publicity and communications with relevant stakeholders. 


and for young people in the trial group, each school will: 


e allocate each young person to a trained reading coach who will work with the young 
person, one to one, for 20 minutes a day, five days a week for 5 weeks. 

e engage an agreed number of young people in reading and seek to improve their 
reading skills 

e enable young people to have access to a wide selection of suitable reading material 

e provide suitable venues for individual reading sessions so that young people feel 
comfortable and able to read aloud 

e encourage young people to read outside of their one to one support time 

e recognise young people’s participation and progression using the MyChoice! awards 
scheme 


Awards Scheme — MyChoice! 


Awards will be provided as an online points reward scheme, 50 points being awarded each day 
to each young person who attends and completes the required work. Young people working on 
the programme will receive points to a maximum value of 1250 points (£12.50 over 5 weeks) as 
part of the Year 6 TextNow award scheme. The points will be awarded according to attendance 
and behaviour on the programme and will be linked to data submitted via the Recording 
template. 
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Young people will work with their coach to choose books online via their personal ‘wish list’ on 
MyChoice! When they have enough points, young people will be able to submit an order to their 
coordinator for purchase. 


In addition, there will be a ‘starter’ library fund which will enable the coordinator to choose a 
library of books for their site. 


Young people participating in the trial group of the TextNow transition pilot will also receive 
additional points to enable them to choose and order books during the summer holidays. 


An overview of the TextNow programme structure: 


Coordinator at each school 


Responsibilities 


Y Recruit & support coaches and allocate to 
young people 


Conduct reading interview with young 
people 


Manage supply of reading material 


Monitor and appraise coaches’ 
performance and record keeping 


Administer MyChoice! and the quality Coach 
assurance system 


Manage child protection Responsibilities 


v¥ Complete the training and provide 
feedback to Unitas 


Plan and deliver regular one to one 
coaching sessions with the young person 


Record daily attendance 


Guide young people in using the online 
award system, MyChoice! 


Young Person 


Responsibilities 
Attend on a daily basis 


Read, with a coach, for 20 minutes each 
day 


Read for a further 20 minutes each day 


Choose books on MyChoice! 


Return any borrowed books 
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Intended outcomes 


The target outcomes are that following participation in the project there will be significant 
increases in the number of young people in the trial group who report that they: 


feel positive about reading to themselves (confidence) 
enjoy reading (enjoyment) 
read daily (engagement) 


and 


who increase their reading scores 


To ensure that TextNow meets the needs of its participants, each school’s programme 
coordinator must commit to participating in the monitoring and evaluation process to enable 
outcomes to be recorded (for example, the completion and emailing of registers and young 
people assessments on the electronic record sheets; coordinator surveys). 


3. 


Input from the site 


Schools must: 


ensure the licence agreement has been signed by an authorised signatory. Schools 
should keep a copy of the agreement and any person involved in delivering the 
programme should be aware of their obligations under the licence agreement 


ensure appropriate numbers of suitable coaches are working on the TextNow 
programme to allow for a one to one ratio with young people 


provide a project coordinator to manage the programme and work closely with the 
coaches, and provide information to all coaches working with TextNow to enable 
them to work effectively with each young person 


ensure coordinators and coaches complete the online training 
identify appropriate young people in Year 6 who are predicted to achieve Level 3, 4c 
or 4b in Reading by the end of Key Stage 2. 


request parental consent for young people's participation 


ensure young people’s attendance and reading for 20 minutes a day, 5 days a week 
for 5 weeks 


record and provide data for monitoring and evaluation purposes 


approve pupils’ book orders on the MyChoice! website and manage distribution of 
‘holiday reading’ books 


encourage young people participating in the trial group of the TextNow transition pilot 
to visit the MyChoice! website and interact with the activities on the site as part of 
their continued independent reading during the summer holidays. 
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4. Input from Unitas 
Unitas will offer each primary school: 


e induction and training for coaches and coordinators through the online training 
resources 

guides for coordinators and coaches (downloadable from the website) 

diaries for coaches (downloadable from the website) 

a choice of your own library of books through MyChoice! 

an online based awards scheme (MyChoice!) 

recording and monitoring facility through electronic record sheets 

information, advice and guidance from Unitas 


5. Training 


It is a requirement that all coaches and coordinators undertake the online training, which 
includes: 


e how to run a TextNow programme, including methods of planning successful 
sessions and encouraging reading 

e how to keep good records to inform future sessions 

e how to manage a successful awards scheme. 


6. Monitoring and Evaluation by Unitas 
The monitoring process will focus on: 


e attendance 
o analysing how many sessions were planned for the learners, how many they 
attended and how long they spent on each session 
e reading outcomes for young people 
o acomparison of attitudes and reading skills at the beginning and end of the pilot. 


The evaluation will analyse the data on attendance and outcomes and may be supplemented 
by qualitative evidence gained through interviews, completion of online surveys and 
compiling case studies. Sites must contribute to and support the monitoring and evaluation 
process. 

7. Independent evaluation 


All schools must participate in the randomised control trial (RCT) and process evaluation that 
will be conducted by Sheffield Hallam University and Queens University, Belfast. 


The purpose of the RCT and process evaluation 
The purpose of the RCT and process evaluation is to answer the following research 


questions: 
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e What is the impact of the TextNow programme on a number of specific reading 
outcomes for participating learners? 
e Is the programme having a differential impact on learners depending on: 
o Their gender? 
o Their socio-economic status? 
e Does the impact of the programme vary significantly with any variations in 
implementation found? 


Evaluation methods and testing 


The RCT will compare the outcomes of the 300 pupils who participate in the TextNow pilot 
with 300 pupils who do not participate. Key stage 2 results will be used as a pre intervention 
test and the New Group Reading test will be administered as a post intervention test to 
measure reading comprehension (in December 2013). The post test will be supplemented by 
The Reading Enjoyment and Confidence scale from the Progress in International Reading 
Literacy Study to measure reading for pleasure. 


The evaluators will match pupil data supplied by schools to data held by the DfE, for example 
gender, ethnicity and key stage 2 results. No pupil names will be used in this analysis as this 
matching will be done using UPNs only. 


The evaluators will also have access to all monitoring and evaluation data collected by 
Unitas and may collect further data through surveys of coordinators and coaches, and 
surveys of, or focus groups with, pupils. 


Anonymised evaluation data will be shared with the Education Endowment Foundation and 
may be used for current and future research. 


Selection of participants 


Secondary schools agreeing to take part in the TextNow pilot will provide a list of all eligible 
pupils who will be starting year 7 in their school in September 2013. Eligible pupils will be 
those who are identified through teacher assessment as not likely to achieve level 4, or likely 
to achieve a fragile level (4b or 4c), in Reading by the end of key stage 2. Pupils likely to 
achieve at or lower than level 2 will not be included in the study. Eligible pupils, once parental 
consent is obtained, will be randomly assigned to either the intervention or the control group. 


Participation required from primary schools to meet evaluation requirements 


e Organising the distribution of information and consent forms to parents and carers of 
eligible pupils and collection of written consent forms. 

e Support for the arrangement of data collection activities such as surveys and focus 
group, if necessary. 


Ethics and data storage 


The project has been checked to ensure it meets ethical standards using the approval 
processes in place at Sheffield Hallam University and Queens University, Belfast. All data will 
be held securely, complying with legal requirements. 
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Achieve 


The Hub 

Secondary Schools Candid Arts 
5 Torrens Street 

London 

EC1V 1NQ 


contact@unitas.uk.net 
To: [Insert name and address of Licensee. Example: 
Name of school (‘the Licensee”)”” 


Date: [Insert date] 
Dear [insert name of authorised representative of the licensee e.g. the Head Teacher] 
Licence 


This letter is to confirm Unitas’ agreement to permit the Licensee (details of which are set 
out above) to use the intellectual property and materials listed in Schedule 1 (“the 
Property’) in connection with the TextNow programme (“the Programme’) for young 
people, as it is used in the TextNow Transition Pilot (“the Pilot”), funded by the Education 
Endowment Foundation. 


Any references to “you” in this letter are references to the Licensee organisation. 
3. Licence 


3.1 Unitas appoints the Licensee as its non-exclusive licensee to use the Property, 
including the registered and unregistered trade marks of Unitas listed as part of the 
Property in Schedule 1 (“the Marks’) in [insert the establishment in which the 
Licensee will use the Property|*' (‘the Site”). The licence lasts for the term of this 
agreement from 1* July 2013 until 31 July 2014, or until terminated under paragraph 
4 below or extended in accordance with paragraph 6.1 below. 


Unitas may as it chooses from time to time provide web or telephone support to the 
Licensee relating to the Property and the Programme. 


3.2 The Property may be used with up to [insert number] of people who are in Year 7 
during the licence period (“young people”), who must complete the Programme by 
the end of the autumn term 2013. This pilot cohort of young people may be exceeded 
only with the written consent of Unitas. 


3.3 If the Licensee wishes to use the Property outside of the Site or to allow another 
organisation to use the Property it may only do so with Unitas’ written consent, not to 


°° The Licensee is the organisation to which the Property is to be licensed to under the MyChoice! 
Reward Scheme. 

*! The establishment will be the geographic area in which the particular TextNow programme service 
is to be provided by the Licensee 
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Ad 


5.1 


5.2 


5.3 


5.4 


5:5 


5.6 


be unreasonably withheld. If Unitas provides its consent for the Licensee to use the 
Property outside of the Site and/or with other organisation(s), the Licensee agrees 
that it remains wholly responsible for its obligations under the Programme and this 
agreement. However, Unitas may enter into a separate arrangement or agreement 
with those other organisation(s). 


Fee 
[There will be no fee payable by the Licensee 
[Insert name of licensee]’s obligations 


The Licensee acknowledges that Unitas is the owner of all rights in and to the 
Property, and agrees: 


to use the Property only for the purposes outlined in Schedule 2; 


to comply at all times with the TextNow Programme Specification at Schedule 3 (“the 
Programme Specification”) and any guidelines Unitas may provide from time to 
time; 


not to bring Unitas or any of Unitas’ funders or partners, or the Property into disrepute 
in any way; 


to obtain Unitas’ prior written approval to all materials (in any media), which bear the 
Marks and to display Marks of funders of the Programme on such materials if 
requested to do so by Unitas; 


to maintain the distinctiveness, reputation and value of the Property and to promptly 
cease any use of the Property that in Unitas’ opinion is not consistent with this at 
Unitas’ request; 


to comply with any Programme evaluation process in accordance with the 
Programme Specification and Unitas’ reasonable instructions issued from time to 
time, including but not limited to:’ 


5.6.1 the prompt facilitation of any visits to the Site by Unitas and/or the evaluation 
team appointed by the Education Endowment Foundation (the Evaluators) 
legislation, guidance and regulations at all times); 


5.6.2 the prompt provision to Unitas on its request of any data relating to the 
Programme delivery, which may include data on both young people and 
coaches (subject to the Licensee complying with its obligations under the Data 
Protection Act 1998 and all other applicable data protection legislation, 
guidance and regulations at all times); 


5.6.3 completion of a pre and post Programme assessment and end of year 
assessment of each young person participating in the Programme; 


5.6.4 completion of a weekly register during the Programme that will be uploaded to 
the Unitas website no later than 12 midday on the Monday of the following 


week; 
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5.8 


6.1 
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5.6.5 provision of access to a computer room to enable all participants (trial and 
control) to take the final reading tests at the same time. 


5.6.6 the participation of the coordinator and coaches in Programme evaluation 
through completion of profiles, online surveys, and interviews, the responses 
to which the Licensee agrees may be used by Unitas and the Evaluators for 
its evaluation, promotional and marketing purposes; 


5.6.7 the participation of young people in completion of surveys and other 
evaluation activities; 


5.6.8 the participation of coaches and young people in the MyChoice! book 
selection and purchase system. Coaches will encourage and facilitate the 
young people to select books on MyChoice! and use the credits they have 
earned to purchase books of their choice. However Unitas bears no 
responsibility for the books provided on MyChoice! and it is the responsibility 
of the coaches and coordinator to monitor and approve book choices 


to ensure that all coaches have undertaken any TextNow online training provided by 
Unitas from time to time; 


subject to paragraph 5.3 below, to comply with all legislation and statutory duties of 
the Licensee insofar as they relate to the Programme, including but not limited to: 


5.8.1 all applicable health and safety legislation; 
5.8.2 all applicable safeguarding legislation; 
5.8.3 all applicable equal opportunities legislation; and 


5.8.4 the Data Protection Act 1998 and all other applicable data protection 
legislation. 


Termination of Licence 


Unitas may terminate this agreement (including the licence) at any time on four 
weeks’ written notice to the Licensee and also reserves the right to terminate it 
immediately if:- 


6.1.1. the Licensee does anything which in Unitas’ opinion brings, or is reasonably 
likely to bring, the Property or reputation of Unitas into disrepute or is inimical 
to the charitable objects of Unitas; and/or 


6.1.2 the Licensee repeatedly breaches any of the terms of this agreement, or 
breaches any material term of this agreement. 


On termination of this agreement, the licence granted under this agreement shall 
cease immediately, meaning the Licensee’s right to use the Property shall cease 
immediately. The Licensee shall not after termination use any materials or intellectual 
property which are or is similar to the Property or might be likely to cause confusion, 
association with or damage to Unitas. 
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Limitation of Unitas’ liability 


The parties acknowledge that the Licensee is solely liable for any loss, cost or 
damage relating to this agreement or the Programme and that Unitas shall not be 
liable for any loss, cost or damage suffered relating to or as a result of the Licensee’s 
participation in the Programme or either party’s compliance with this agreement 
(except in the case of death or personal injury caused by Unitas’ negligence in which 
case no limitation applies). Nothing in this paragraph limits or excludes any liability for 
fraud. 


The Licensee agrees to reimburse Unitas in respect of any losses, expenses or 
damage Unitas may incur (including direct, indirect, special and consequential loss, 
and any loss of reputation or loss of profits, including reasonable legal costs and 
disbursements) as a result of any breach by the Licensee of any term of this 
agreement or as a result of any of its actions or omissions, relating to Unitas and/or 
the Property. 


It is the Licensee’s sole responsibility to ensure the Licensee complies with its 
statutory duties (including but not limited to those listed at paragraph 3.8 above) and 
Unitas shall not be responsible in any way for the Licensee’s failure to comply with 
such duties. 


General 


Subject to any amendments or variations to the Property in accordance with 
Schedule 1 or permissions granted under paragraph 1 above, this agreement 
constitutes the entire agreement between Unitas and the Licensee and shall not be 
varied or amended except by written agreement between the parties. 


The Licensee agrees that all data provided to Unitas may be shared with the 
Evaluators. 


The Licensee shall not assign or sub-licence any of its rights under this agreement 
without the prior written consent of Unitas. 


This agreement shall not constitute the parties as partners nor is the Licensee entitled 
to act as nor represent itself as agent for Unitas, nor to pledge Unitas’ credit. 


This agreement is governed by and shall be construed in accordance with the laws of 
England and Wales and the parties irrevocably submit to the exclusive jurisdiction of 
the courts of England and Wales. 


If you have any queries or concerns about agreeing to these terms, please feel free to call 
Ella Bourne on 07557 767721. Otherwise, please confirm the terms are agreed by signing, 
dating and returning to Unitas the enclosed copy of this letter by post, fax, or email showing a 
scanned signature. 
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With many thanks. 


Yours sincerely, 


my 


Signed for and on behalf of Unitas 
Name: Martin Stephenson 
Position: Chief Executive Officer 


Date: 


Signed for and on behalf of [name of Licensee]”” 
Name: 
Position: 


Date: 


?? This agreement must be signed by an authorised signatory only 
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Schedule 1 


The Property 


10. 


Unitas shall provide the following material in such form and in such volume as it shall 
in its sole discretion decide from time to time. The Property may be changed or varied 
from time to time on written notification to the Licensee by Unitas. 


The materials 


Credits to enable the coordinator to obtain a starter library of books for use on the 
TextNow programme 

Access to online training for TextNow coaches and coordinators including 
downloadable resources. 

Group based training resources 

Access to an online quality assurance system 

Access to MyChoice!, an online awards scheme, and up to 2500 credits per young 
person (credits to be generated according to attendance data as submitted through 
the online quality assurance system) 

For this Transition Pilot additional MyChoice! credits will also be made available to 
each young person in the trial group to enable them to choose books for independent 
reading between the end of their TextNow programme and the end of the academic 
year 2013-2014. 

Young people participating in the Control group of the pilot will receive a ‘Thank you’ 
award of vouchers to buy books after the December assessment. 


The Marks 

‘TextNow’ 

UK trade mark number 2576568. 
Status: registered. 


“Unitas” 


UK trade mark number: 2515758. 
Status: registered. 


Schedule 2 


The Licensee shall use the Property: 


in accordance with the agreement; 

in accordance with the requirements of the pilot 

in line with Unitas’ instructions from time to time; 

for educational purposes only; and 

in accordance with the Programme Specification at Schedule 3. 
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CreateAchieve 
TextNow Programme Specification 


(for Secondary schools taking part in the TextNow Transition Pilot) 
8. Introduction 


The TextNow transition pilot is a randomised controlled trial evaluation of the ‘Unitas: 
TextNow Transition Programme’ for improving reading outcomes among young people in 
transition between Primary and Secondary school 


Unitas will deliver their TextNow Transition Programme to 300 of the 600 participating pupils 
between May and December 2013 (the trial group). The remaining 300 young people will 
receive no intervention during the time period (the control group). 


Pupils in both groups will be assessed at various points throughout the trial to provide 
comparative data for evaluation. It is therefore extremely important that pilot schools maintain 
the integrity of the study by ensuring that control group pupils complete the assessments 
according to the pilot schedule and that they do not receive any alternative intervention 
during the period of the trial. 


This document outlines the TextNow programme for young people who have been identified 
as participating in the trial group of the TextNow transition pilot and who are likely to benefit 
from increasing their motivation to read and their enjoyment of, and proficiency in, reading. 


It is designed to enable organisations delivering TextNow to gain a clear understanding of: 


e what is involved in delivering TextNow 
e the intended benefits 
e the obligations on the part of the host organisation. 


TextNow aims to help young people to be more engaged with reading and to read more 
fluently. This can contribute towards preventing social disadvantage and social exclusion and 
enable young people to more fully engage with their education and training. TextNow was 
developed to work with the target group of young people aged 10-19 years old. For the 
purposes of the TextNow transition pilot the selected pupils will be from Year 6 moving into 
Year 7. 


The TextNow transition pilot involves trained reading coaches working with individual young 
people in Year 7 for daily 20 minutes sessions over ten weeks, (following a similar five week 
programme at the end of Year 6). A project coordinator, designated by the school manages 
the delivery of the programme at each site. 


9. Programme Structure 


The objectives of the TextNow programme are to: 
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1. increase young people’s reading. 

2. enhance young people’s enjoyment of reading. 

3. improve the reading skills of participating young people, including their 
understanding of what they read. 

Each school will: 


e identify young people and their feeder primary schools who are eligible to participate 
in this pilot 

e complete all the appropriate records for every participating young person and return 
them regularly to the Unitas team where required 

e participate fully in the evaluation process, through completion of surveys and 
interviews 

e establish close links with parents/carers to ensure that all those involved are working 
towards the same goals 

e acknowledge Unitas in all publicity and communications with relevant stakeholders. 


and for young people in the trial group 


e engage an agreed number of young people in reading and seek to improve their 
reading skills 

e allocate each young person to a trained reading coach who will work with the young 
person, one to one, for 20 minutes a day, five days a week for 10 weeks 

e provide suitable venues for individual reading sessions so that young people feel 
comfortable and able to read aloud 

e enable young people to have access to a wide selection of suitable reading material 

e encourage young people to read outside of their one to one support time 

e recognise young people’s participation and progression using the MyChoice! awards 
scheme 


Awards Scheme — MyChoice! 


Awards will be provided as an online points reward scheme, 50 points being awarded each day 
to each young person who attends and completes the required work. Young people working on 
the programme will receive points to a maximum value of 2500 points (£25.00 over 10 weeks) as 
part of the Year 7 TextNow award scheme. The points will be awarded according to attendance 
and behaviour on the programme and will be linked to data submitted via the online quality 
assurance system. 


Young people will work with their coach to choose books online via their personal ‘wish list’ on 
MyChoice! When they have enough points, young people will be able to submit an order to their 
coordinator for purchase. 


In addition, there will be a ‘starter’ library fund which will enable the coordinator to choose a 
library of books for their site. 


82 


Young people participating in the trial group of the TextNow transition pilot will also, after the 
completion of the daily reading sessions, periodically receive additional points to enable them 
to choose and order books up to the end of the academic year 2013-2014. 
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An overview of the TextNow programme structure: 


Coordinator at each school 


Responsibilities 


¥ Recruit & support coaches and allocate to 
young people 


Conduct reading interview with young 
people 


Manage supply of reading material 


Monitor and appraise coaches’ 
performance and record keeping 


Conduct young people assessments and 
reviews 


Administer MyChoice! and the quality 
assurance system Coach 


Manage child protection 


Responsibilities 


v¥ Complete the training and provide 
feedback to Unitas 


Plan and deliver regular one to one 
coaching sessions with the young person 


Record daily attendance 
Young Person 


Guide young people in using the online 
award system, MyChoice! 


Arrange final reviews 
Responsibilities 


v¥_ Attend on a daily basis 


Y Read, with a coach, for 20 minutes each 
day 


Read for a further 20 minutes each day 


Choose books on MyChoice! 


Return any borrowed books 
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10. Intended outcomes 


The target outcomes are that following participation in the project by the trial group there will 
be significant increases in the number of young people who report that they: 


feel positive about reading to themselves (confidence) 
enjoy reading (enjoyment) 
read daily (engagement) 


and 


who increase their reading scores 


To ensure that TextNow meets the needs of its participants, each school’s programme 
coordinator must commit to participating in the monitoring and evaluation process to enable 
outcomes to be recorded (for example, the completion and uploading of registers and young 
people assessments onto the online quality assurance system; coach surveys). 


11. Input from the site 


Schools must: 


ensure the licence agreement has been signed by an authorised signatory. Schools 
should keep a copy of the agreement and any person involved in delivering the 
programme should be aware of their obligations under the licence agreement 

ensure appropriate numbers of suitable coaches are working on the TextNow 
programme to allow for a one to one ratio with young people 

provide a project coordinator to manage the programme and work closely with the 
coaches 

ensure coordinators and coaches complete the online training 

Identify appropriate young people in Year 6 who are predicted to achieve Level 3, 4c 
or 4b in Reading by the end of Key Stage 2. 

provide information to all coaches working with TextNow to enable them to work 
effectively with each young person 

ensure young people’s attendance and reading for 20 minutes a day, 5 days a week 
for 10 weeks 

record and provide data for monitoring and evaluation 

approve pupils’ book orders on the MyChoice! website 

encourage young people participating in the trial group of the TextNow transition pilot 
to visit the MyChoice! website and interact with the activities on the site as part of 
their continued independent reading. 


12. Input from Unitas 


, Unitas will offer each secondary school: 


induction and training for coaches and coordinators through the online training 
resources 
guides for coordinators and coaches (downloadable from the website) 
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diaries for coaches (downloadable from the website) 

a choice of your own library of books through MyChoice! 

an online based awards scheme (MyChoice!) 

recording and monitoring facility through an online quality assurance system 
information, advice and guidance from Unitas 

performance report. 


13. Training 


It is a requirement that all coaches and coordinators undertake the online training, which 
includes: 


e how to run a TextNow programme, including methods of planning successful 
sessions and encouraging reading 

e how to keep good records to inform future sessions 

e how to manage a successful rewards scheme. 


14. Monitoring and Evaluation by Unitas 
The monitoring process will focus on: 


e attendance 
o analysing how many sessions were planned for the learners, how many they 
attended and how long they spent on each session 
e reading outcomes for young people 
o acomparison of attitudes and reading skills at the beginning, during and end of 
the pilot. 


The evaluation will analyse the data on attendance and outcomes and may be supplemented 
by qualitative evidence gained through interviews, completion of online surveys and 
compiling case studies. Sites must contribute to and support the monitoring and evaluation 
process. 


15. Independent evaluation 


All schools must participate in the randomised control trial (RCT) and process evaluation that 
will be conducted by Sheffield Hallam University and Queens University, Belfast. 


The purpose of the RCT and process evaluation 


The purpose of the RCT and process evaluation is to answer the following research 
questions: 


e What is the impact of the TextNow programme on a number of specific reading 
outcomes for participating learners? 
e ls the programme having a differential impact on learners depending on: 
o Their gender? 
o Their socio-economic status? 
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Does the impact of the programme vary significantly with any variations in 
implementation found? 
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Evaluation methods and testing 


The RCT will compare the outcomes of the 300 pupils who participate in the TextNow pilot 
with 300 pupils who do not participate. Key stage 2 results will be used as a pre intervention 
test and the New Group Reading test will be administered as a post intervention test to 
measure reading comprehension. The post test will be supplemented by The Reading 
Enjoyment and Confidence scale from the Progress in International Reading Literacy Study 
to measure reading for pleasure. 


The evaluators will match pupil data supplied by schools to data held by the DfE, for example 
gender, ethnicity and key stage 2 results. No pupil names will used in this analysis as 
matching will be done using UPNs only. 


The evaluators will also have access to all monitoring and evaluation data collected by 
Unitas and may collect further data through surveys of coordinators and coaches, and 
surveys of, or focus groups with, pupils. 


Anonymised evaluation data will be shared with the Education Endowment Foundation and 
may be used for current and future research. 


Participation required from schools to meet the evaluation requirements 


e Provision of a list of eligible pupils' names, their UPN and primary school as soon as 
possible after admissions are confirmed. A separate list of KS2 teacher assessments 
for reading matched to UPN only. 

e Making organisational arrangements for the administration of the post test. This will 
require all pupils in the intervention or control groups in the school to be brought 
together to undertake the post test. The test will be conducted electronically so will 
require computer facilities. 

e Support for the arrangement of other data collection activities e.g. surveys and focus 
group, if necessary. 


Ethics and data storage 


The project has been checked to ensure it meets ethical standards using the approval 
processes in place at Sheffield Hallam University and Queens University, Belfast. All data will 
be held securely, complying with legal requirements. 


Appendix VII. Security of finding 
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The final security rating for this trial is 3 @. This means that findings are moderately secure. 


The TextNow Transition trial was originally designed as a well-powered, two-armed effectiveness trial 
with minimum detectable effect of 0.19 that could have achieved the maximum rating of 5 @. However, 
the trial had relatively high attrition of 22%. Delivery and marking of post-testing was blind to treatment 
allocation. There were problems with implementation fidelity, some evidence of non-compliance (with 
three control children being given the treatment) and some evidence of resentful demoralisation in the 
intervention group. 
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Appendix VIII Cost rating 


Cost ratings are based on the approximate cost per pupil of implementing the intervention over one 
year. Cost ratings are awarded using the following criteria. 


Cost 
£ 
£E 


Description 
Very low: less than £80 per pupil per year. 
Low: up to about £170 per pupil per year. 


Moderate: up to about £700 per pupil per year. 


High: up to £1,200 per pupil per year. 
Very high: over £1,200 per pupil per year. 
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You may re-use this document/publication (not including logos) free of charge in any format or 
medium, under the terms of the Open Government Licence v2.0. 


To view this licence, visit www.nationalarchives.gov.uk/doc/open-government-licence/version/2 
or email: psi@nationalarchives.gsi.gov.uk 


Where we have identified any third party copyright information you will need to obtain permission from 
the copyright holders concerned. The views expressed in this report are the authors’ and do not 
necessarily reflect those of the Department for Education. 


This document is available for download at www.educationendowmentfoundation.org.uk. 
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The Education Endowment Foundation 

9th Floor, Millbank Tower 

21-24 Millbank 

London 

SW1P 4QP 
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